What are they, where are they, and are they best for you?
Input and output (I/O) operations discuss with the transfer of knowledge between a pc’s predominant memory and various peripherals. Storage peripherals comparable to HDDs and SSDs have particular performance characteristics when it comes to latency, throughput, and rate which might influence the performance of the pc system they power. Extrapolating, the performance and design of distributed and cloud based Data Storage depends upon that of the medium. This text is meant to be a bridge between Data Science and Storage Systems: 1/ I’m sharing a number of datasets of varied sources and sizes which I hope can be novel for Data Scientists and a couple of/ I’m bringing up the potential for advanced analytics in Distributed Systems.
Storage access traces are “a treasure trove of data for optimizing cloud workloads.” They’re crucial for capability planning, data placement, or system design and evaluation, fitted to modern applications. Diverse and up-to-date datasets are particularly needed in academic research to review novel and unintuitive access patterns, help the design of recent hardware architectures, recent caching algorithms, or hardware simulations.
Storage traces are notoriously difficult to search out. The SNIA website is one of the best known “repository for storage-related I/O trace files, associated tools, and other related information” but many traces don’t comply with their licensing or upload format. Finding traces becomes a tedious means of scanning the tutorial literature or attempting to generate one’s own.
Popular traces that are easier to search out are likely to be outdated and overused. Traces older than 10 years shouldn’t be utilized in modern research and development as a result of changes in application workloads and hardware capabilities. Also, an over-use of specific traces can bias the understanding of real workloads so it’s really useful to make use of traces from multiple independent sources when possible.
This post is an organized collection of recent public traces I discovered and used. In the primary part I categorize them by the extent of abstraction they represent within the IO stack. Within the second part I list and discuss some relevant datasets. The last part is a summary of all with a private view on the gaps in storage tracing datasets.
I distinguish between three varieties of traces based on data representation and access model. Let me explain. A user, at the appliance layer, sees data stored in files or objects that are accessed by a wide range of abstract operations comparable to open or append. Closer to the media, the info is stored in a continuous memory address space and accessed as blocks of fixed size which can only be read or written. At the next abstraction level, inside the appliance layer, we may additionally have an information presentation layer which can log access to data presentation units, which could also be, for instance, rows composing tables and databases, or articles and paragraphs composing news feeds. The access could also be create table, or post article.
While traces may be taken anywhere within the IO stack and contain information from multiple layers, I’m selecting to structure the next classification based on the Linux IO stack depicted below.
Block storage traces
The info in these traces is representative of the operations on the block layer. In Linux, this data is often collected with blktrace (and rendered readable with blkparse), iostat, or dtrace. The traces contain information concerning the operation, the device, CPU, process, and storage location accessed. The primary trace listed is an example of blktrace output.
The standard information generated by tracing programs could also be too detailed for evaluation and publication purposes and it is commonly simplified. Typical public traces contain operation, offset, size, and sometimes timing. At this layer the operations are only read and write. Each operation accesses the address starting at offset and is applied to a continuous size of memory laid out in variety of blocks (4KiB NTFS). For instance, a trace entry for a read operation comprises the address where the read starts (offset), and the variety of blocks read (size). The timing information may contain the time the request was issued (start time), the time it was accomplished (end time), the processing in between (latency), and the time the request waited (queuing time).
Available traces sport different features, have wildly different sizes, and are the output of quite a lot of workloads. Choosing the correct one will rely upon what one’s in search of. For instance, trace replay only needs the order of operations and their size; For performance evaluation timing information is required.
Object storage traces
At the appliance layer, data is positioned in files and objects which could also be created, opened, appended, or closed, after which discovered via a tree structure. From an user’s standpoint, the storage media is decoupled, hiding fragmentation, and allowing random byte access.
I’ll group together file and object traces despite a subtle difference between the 2. Files follow the file system’s naming convention which is structured (typically hierarchical). Often the extension suggests the content type and usage of the file. Alternatively, objects are utilized in large scale storage systems coping with vast amounts of diverse data. In object storage systems the structure shouldn’t be intrinsic, as an alternative it’s defined externally, by the user, with specific metadata files managed by their workload.
Being generated inside the appliance space, typically the results of an application logging mechanism, object traces are more diverse when it comes to format and content. The knowledge recorded could also be more specific, for instance, operations may also be delete, copy, or append. Objects typically have variable size and even the identical object’s size may vary in time after appends and overwrites. The object identifier generally is a string of variable size. It could encode extra information, for instance, an extension that tells the content type. Other meta-information may come from the range accessed, which can tell us, for instance, whether the header, the footer or the body of a picture, parquet, or CSV file was accessed.
Object storage traces are higher fitted to understanding user access patterns. By way of block access, a video stream and a sequential read of a complete file generate the identical pattern: multiple sequential IOs at regular time intervals. But these trace entries needs to be treated otherwise if we’re to replay them. Accessing video streaming blocks must be done with the identical time delta between them, whatever the latency of every individual block, while reading all the file needs to be asap.
Access traces
Specific to every application, data could also be abstracted further. Data units could also be instances of a category, records in a database, or ranges in a file. A single data access may not even generate a file open or a disk IO if caching is involved. I select to incorporate such traces because they might be used to know and optimize storage access, and specifically cloud storage. For instance, the access traces from Twitter’s Memcache are useful in understanding popularity distributions and subsequently could also be useful for data formatting and placement decisions. Often they are not storage traces per se, but they may be useful within the context of cache simulation, IO reduction, or data layout (indexing).
Data format in these traces may be much more diverse as a result of a brand new layer of abstraction, for instance, by tweet identifiers in Memcached.
Let’s take a look at a number of traces in each of the categories above. The list details among the newer traces — no older than 10 years — and it’s under no circumstances exhaustive.
Block traces
YCSB RocksDB SSD 2020
These are SSD traces collected on a 28-core, 128 GB host with two 512 GB NVMe SSD Drives, running Ubuntu. The dataset is a results of running the YCSB-0.15.0 benchmark with RocksDB.
The primary SSD stores all blktrace output, while the second hosts YCSB and RocksDB. YCSB Workload A consists of fifty% reads and 50% updates of 1B operations on 250M records. Runtime is 9.7 hours, which generates over 352M block I/O requests on the file system level writing a complete of 6.8 TB to the disk, with a read throughput of 90 MBps and a write throughput of 196 MBps.
The dataset is small in comparison with all others within the list, and limited when it comes to workload, but an awesome place to begin as a result of its manageable size. One other profit is reproducibility: it uses open source tracing tools and benchmarking beds atop a comparatively inexpensive hardware setup.
Format: These are SSD traces taken with blktrace and have the everyday format after parsing with blkparse: [Device Major Number,Device Minor Number] [CPU Core ID] [Record ID] [Timestamp (in nanoseconds)] [ProcessID] [Trace Action] [OperationType] [SectorNumber + I/O Size] [ProcessName]
259,2 0 1 0.000000000 4020 Q R 282624 + 8 [java]
259,2 0 2 0.000001581 4020 G R 282624 + 8 [java]
259,2 0 3 0.000003650 4020 U N [java] 1
259,2 0 4 0.000003858 4020 I RS 282624 + 8 [java]
259,2 0 5 0.000005462 4020 D RS 282624 + 8 [java]
259,2 0 6 0.013163464 0 C RS 282624 + 8 [0]
259,2 0 7 0.013359202 4020 Q R 286720 + 128 [java]
Where to search out it: http://iotta.snia.org/traces/block-io/28568
License: SNIA Trace Data Files Download License
Alibaba Block Traces 2020
The dataset consists of “block-level I/O requests collected from 1,000 volumes, where each has a raw capability from 40 GiB to five TiB. The workloads span diverse varieties of cloud applications. Each collected I/O request specifies the quantity number, request type, request offset, request size, and timestamp.”
Limitations (from the tutorial paper)
- the traces don’t record the response times of the I/O requests, making them unsuitable for latency evaluation of I/O requests.
- the precise applications running atop usually are not mentioned, so that they can’t be used to extract application workloads and their I/O patterns.
- the traces capture the access to virtual devices, so that they usually are not representative of performance and reliability (e.g., data placement and failure statistics) for physical block storage devices.
A drawback of this dataset is its size. When uncompressed it leads to a 751GB file which is difficult to store and manage.
Format: device_id,opcode,offset,length,timestamp
device_idID of the virtual disk,uint32opcodeEither of ‘R’ or ‘W’, indicating this operation is read or writeoffsetOffset of this operation, in bytes,uint64lengthLength of this operation, in bytes,uint32timestampTimestamp of this operation received by server, in microseconds,uint64
419,W,8792731648,16384,1577808144360767
725,R,59110326272,360448,1577808144360813
12,R,350868463616,8192,1577808144360852
725,R,59110686720,466944,1577808144360891
736,R,72323657728,516096,1577808144360996
12,R,348404277248,8192,1577808144361031
Moreover, there may be an additional file containing each virtual device’s id device_id with its total capability.
Where to search out it: https://github.com/alibaba/block-traces
License: CC-4.0.
Tencent Block Storage 2018
This dataset consists of “216 I/O traces from a warehouse (also called a failure domain) of a production cloud block storage system (CBS). The traces are I/O requests from 5584 cloud virtual volumes (CVVs) for ten days (from Oct. 1st to Oct. tenth, 2018). The I/O requests from the CVVs are mapped and redirected to a storage cluster consisting of 40 storage nodes (i.e., disks).”
Limitations:
- Timestamps are in seconds, a granularity too little for determining the order of operations. As a consequence many requests appear as if issued at the identical time. This trace is subsequently unsuitable for queuing evaluation.
- There isn’t a latency information concerning the duration of every operation, making the trace unsuitable for latency performance, queuing analytics.
- No extra details about each volume comparable to total size.
Format: Timestamp,Offset,Size,IOType,VolumeID
Timestampis the Unix time the I/O was issued in seconds.Offsetis the starting offset of the I/O in sectors from the beginning of the logical virtual volume. 1 sector = 512 bytesSizeis the transfer size of the I/O request in sectors.IOTypeis “Read(0)”, “Write(1)”.VolumeIDis the ID variety of a CVV.
1538323200,12910952,128,0,1063
1538323200,6338688,8,1,1627
1538323200,1904106400,384,0,1360
1538323200,342884064,256,0,1360
1538323200,15114104,8,0,3607
1538323200,140441472,32,0,1360
1538323200,15361816,520,1,1371
1538323200,23803384,8,0,2363
1538323200,5331600,4,1,3171
Where to search out it: http://iotta.snia.org/traces/parallel/27917
License: NIA Trace Data Files Download License
K5cloud Traces 2018
This dataset comprises traces from virtual cloud storage from the FUJITSU K5 cloud service. The info is gathered during per week, but not constantly because “ at some point’s IO access logs often consumed the storage capability of the capture system.” There are 24 billion records from 3088 virtual storage nodes.
The info is captured within the TCP/IP network between servers running on hypervisor and storage systems in a K5 data center in Japan. The info is split between three datasets by each virtual storage volume id. Each virtual storage volume id is exclusive in the identical dataset, while each virtual storage volume id shouldn’t be unique between the several datasets.
Limitations:
- There isn’t a latency information, so the traces can’t be used for performance evaluation.
- The overall node size is missing, but it may be approximated from the utmost offset accessed within the traces.
- Some applications may require an entire dataset, which makes this one unsuitable as a result of missing data.
The fields within the IO access log are: ID,Timestamp,Type,Offset,Length
IDis the virtual storage volume id.Timestampis the time elapsed from the primary IO request of all IO access logs in seconds, but with a microsecond granularity.Typeis R(Read) or (W)Write.Offsetis the starting offset of the IO access in bytes from the beginning of the virtual storage.Lengthis the transfer size of the IO request in bytes.
1157,3.828359000,W,7155568640,4096
1157,3.833921000,W,7132311552,8192
1157,3.841602000,W,15264690176,28672
1157,3.842341000,W,28121042944,4096
1157,3.857702000,W,15264718848,4096
1157,9.752752000,W,7155568640,4096
Where to search out it: http://iotta.snia.org/traces/parallel/27917
License: CC-4.0.
Object traces
Server-side I/O request arrival traces 2019
This repository comprises two datasets for IO block traces with additional file identifiers: 1/ parallel file systems (PFS) and a couple of/ I/O nodes.
Notes:
- The access patterns are resulting from MPI-IO test benchmark ran atop of Grid5000, a big scale test bed for parallel and High Performance Computing (HPC). These traces usually are not representative of general user or cloud workloads but as an alternative specific to HPC and parallel computing.
- The setup for the PFS scenario uses Orange FS as file system and for the IO nodes I/O Forwarding Scalability Layer(IOFSL). In each cases the scheduler was set to AGIOS I/O scheduling library. This setup is maybe too specific for many use cases targeted by this text and has been designed to reflect some proposed solutions.
- The hardware setup for PFS consists of our server nodes with 600 GB HDDs each and 64 client nodes. For IO nodes, it has 4 server nodes with similar disk configuration in a cluster, and 32 clients in a unique cluster.
Format: The format is barely different for the 2 datasets, an artifact of various file systems. For IO nodes, it consists of multiple files, each with tab-separated values Timestamp FileHandle RequestType Offset Size. A peculiarity is that reads and writes are in separate files named accordingly.
Timestampis a number representing the interior timestamp in nanoseconds.FileHandleis the file handle in hexadecimal of size 64.RequestTypeis the form of the request, inverted, “W” for reads and “R” for writes.Offsetis a number giving the request offset in bytesSizeis the dimensions of the request in bytes.
265277355663 00000000fbffffffffffff0f729db77200000000000000000000000000000000 W 2952790016 32768
265277587575 00000000fbffffffffffff0f729db77200000000000000000000000000000000 W 1946157056 32768
265277671107 00000000fbffffffffffff0f729db77200000000000000000000000000000000 W 973078528 32768
265277913090 00000000fbffffffffffff0f729db77200000000000000000000000000000000 W 4026531840 32768
265277985008 00000000fbffffffffffff0f729db77200000000000000000000000000000000 W 805306368 32768
The PFS scenario has two concurrent applications, “app1” and “app2”, and its traces are inside a folder named accordingly. Each row entry has the next format: [Different from the above are:
RequestTypeis 0 for reads and 1 for writesQueueElementisn’t used and I imagine it’s an artifact of the tracing tool.
[D 01:11:03.153625] REQ SCHED SCHEDULING, handle: 5764607523034233445, queue_element: 0x12986c0, type: 1, offset: 369098752, len: 1048576
[D 01:11:03.153638] REQ SCHED SCHEDULING, handle: 5764607523034233445, queue_element: 0x1298e30, type: 1, offset: 268435456, len: 1048576
[D 01:11:03.153651] REQ SCHED SCHEDULING, handle: 5764607523034233445, queue_element: 0x1188b80, type: 1, offset: 0, len: 1048576
[D 01:11:03.153664] REQ SCHED SCHEDULING, handle: 5764607523034233445, queue_element: 0xf26340, type: 1, offset: 603979776, len: 1048576
[D 01:11:03.153676] REQ SCHED SCHEDULING, handle: 5764607523034233445, queue_element: 0x102d6e0, type: 1, offset: 637534208, len: 1048576
Where to search out it: https://zenodo.org/records/3340631#.XUNa-uhKg2x
License: CC-4.0.
IBM Cloud Object Store 2019
These are anonymized traces from the IBM Cloud Object Storage service collected with the first goal to review data flows to the item store.
The dataset consists of 98 traces containing around 1.6 Billion requests for 342 Million unique objects. The traces themselves are about 88 GB in size. Each trace comprises the REST operations issued against a single bucket in IBM Cloud Object Storage during a single week in 2019. Each trace comprises between 22,000 to 187,000,000 object requests. All of the traces were collected in the course of the same week in 2019. The traces contain all data access requests issued over per week by a single tenant of the service. Object names are anonymized.
Some characteristics of the workload have been published on this paper, although the dataset used was larger:
- The authors were “in a position to discover among the workloads as SQL queries, Deep Learning workloads, Natural Language Processing (NLP), Apache Spark data analytic, and document and media servers. But lots of the workloads’ types remain unknown.”
- “A overwhelming majority of the objects (85%) within the traces are smaller
than a megabyte, Yet these objects only account for 3% of the
of the stored capability.” This made the info suitable for a cache evaluation.
Format: The timestamp is the variety of milliseconds from the purpose where we began collecting the traces.
1219008 REST.PUT.OBJECT 8d4fcda3d675bac9 1056
1221974 REST.HEAD.OBJECT 39d177fb735ac5df 528
1232437 REST.HEAD.OBJECT 3b8255e0609a700d 1456
1232488 REST.GET.OBJECT 95d363d3fbdc0b03 1168 0 1167
1234545 REST.GET.OBJECT bfc07f9981aa6a5a 528 0 527
1256364 REST.HEAD.OBJECT c27efddbeef2b638 12752
1256491 REST.HEAD.OBJECT 13943e909692962f 9760
Where to search out it: http://iotta.snia.org/traces/key-value/36305
License: SNIA Trace Data Files Download License
Access traces
Wiki Analytics Datasets 2019
The wiki dataset comprises data for 1/ upload (image) web requests of Wikimedia and a couple of/ text (HTML pageview) web requests from one CDN cache server of Wikipedia. The mos recent dataset, from 2019 comprises 21 upload data files and 21 text data files.
Format: Each upload data file, denoted cache-u, comprises exactly 24 hours of consecutive data. These files are each roughly 1.5GB in size and hold roughly 4GB of decompressed data each.
This dataset is the results of a single form of workload, which can limit the applicability, however it is large and complete, which makes an excellent testbed.
Each decompressed upload data file has the next format: relative_unix hashed_path_query image_type response_size time_firstbyte
relative_unix: Seconds since start timestamp of dataset, inthashed_path_query: Salted hash of path and query of request, bigintimage_type: Image type from Content-Type header of response, stringresponse_size: Response size in bytes, inttime_firstbyte: Seconds to first byte, double
0 833946053 jpeg 9665 1.85E-4
0 -1679404160 png 17635 2.09E-4
0 -374822678 png 3333 2.18E-4
0 -1125242883 jpeg 4733 1.57E-4
Each text data file, denoted cache-t, comprises exactly 24 hours of consecutive data. These files are each roughly 100MB in size and hold roughly 300MB of decompressed data each.
Each decompressed upload data file has the next format: relative_unix hashed_host_path_query response_size time_firstbyte
4619 540675535 57724 1.92E-4
4619 1389231206 31730 2.29E-4
4619 -176296145 20286 1.85E-4
4619 74293765 14154 2.92E-4
Where to search out it: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Caching
License: CC-4.0.
Memcached 2020
This dataset comprises one-week-long traces from Twitter’s in-memory caching (Twemcache / Pelikan) clusters. The info comes from 54 largest clusters in Mar 2020, Anonymized Cache Request Traces from Twitter Production.
Format: Each trace file is a csv with the format: timestamp,anonymized key,key size,value size,client id,operation,TTL
timestamp: the time when the cache receives the request, in secanonymized key: the unique key with anonymization where namespaces are preserved; for instance, if the anonymized secret isnz:u:eeW511W3dcH3de3d15ec, the primary two fieldsnzanduare namespaces, note that the namespaces usually are not necessarily delimited by:, different workloads use different delimiters with different variety of namespaces.key size: the dimensions of key in bytesvalue size: the dimensions of value in bytesclient id: the anonymized clients (frontend service) who sends the requestoperation: one in all get/gets/set/add/replace/cas/append/prepend/delete/incr/decrTTL: the time-to-live (TTL) of the item set by the client, it’s 0 when the request shouldn’t be a write request.
0,q:q:1:8WTfjZU14ee,17,213,4,get,0
0,yDqF:3q:1AJrrJ1nnCJKKrnGx1A,27,27,5,get,0
0,q:q:1:8WTw2gCuJe8,17,720,6,get,0
0,yDqF:vS:1AJr9JnArxCJGxn919K,27,27,7,get,0
0,yDqF:vS:1AJrrKG1CAnr1C19KxC,27,27,8,get,0
License: CC-4.0.
In case you’re still here and haven’t gone diving into one in all the traces linked above it could be since you haven’t found what you’re in search of. There are a number of gaps that current storage traces have yet to fill:
- Multi-tenant Cloud Storage: Large cloud storage providers store among the most wealthy datasets on the market. Their workload reflects a big scale systems’ architecture and is the results of a various set of applications. Storage providers are also extra cautious with regards to sharing this data. There’s little or no financial incentive to share data with the general public and a fear of unintended customer data leaks.
- Full stack. Each layer within the stack offers a unique view on access patterns, none alone being enough to know cause-and-effect relationships in storage systems. Optimizing a system to suit modern workloads requires a holistic view of the info access which usually are not publicly available.
- Distributed tracing. Most data is nowadays accessed remotely and managed in large scale distributed systems. Many components and layers (comparable to indexes or caching) will alter the access patterns. In such an environment, end-to-end means tracing a request across several components in a fancy architecture. This data may be truly helpful for designing large scale systems but, at the identical time, could also be too specific to the system inspected which, again, limits the inducement to publish it.
- Data quality. The traces above have limitations as a result of the extent of detail they represent. As we’ve got seen, some have missing data, some have large granularity time stamps, others are inconveniently large to make use of. Cleansing data is a tedious process which limits the dataset publishing nowadays.