FAQ

dStor is a cloud storage service that stores data on a distributed network of computers around the world. Its aim is to provide a lower cost alternative to leading centralized cloud storage providers such as Amazon Web Services and Microsoft Azure while retaining their high data permanence and rapid access time. dStor also aims to provide operators of computers a way to make additional incremental income from their infrastructure by operating storage nodes that store and deliver data.

dStor operates a computer network consisting mainly of storage nodes that store files and deliver them to users, and gateway nodes that monitor the health and usage of the network for routing and accounting purposes. There is also an accounting system integrated into dStor to ensure users are correctly billed for their usage and storage node operators are correctly paid for their service.

When an end user wants to access data in the system, their computer sends a request to one of the dStor gateway nodes (assigned round-robin style) which interprets the request based on the location of the requesting IP address and seeks to reflect that request quickly to one of the servers in the same region with the highest availability at that moment based on the gateway node’s ongoing monitoring.

Once the end user’s request is reflected to a storage node in the system, it asks for the requested data asset. If the file exists in the server’s storage or cache, it is immediately served. If the storage node does not have the file, it requests it from another node in the network and once it receives the file, it also caches it for a period of time to hasten future requests.

When a user wants to add data into the system for storage, they make a request to the gateway nodes using the API key associated with their account. Once the system ingests the data, it creates a unique filehash of it that serves as a resource locator. This is how end users or their computers access this data in the future. The gateway node forwards the data to a number of storage nodes (generally 16-20, depending on how many nodes are currently in the network) where they are “pinned” to ensure they are not deleted.

The dStor accounting subsystem monitors all file access and delivery and regularly polls storage nodes to ensure they have their pinned data, which they receive pay for. If pinned files are missing from the system, dStor requests another storage node to pin them instead and the previous node loses future income from storing that data seed. The accounting system calculates invoices for users and payment for storage node operators based on usage.

Maintaining a table of ideal routing within a network is very demanding and therefore time consuming. To be able to process a high volume of requests, dStor gateway nodes monitor the general health of the nodes and aim to very quickly determine a suitable storage node with high resource availability. While an ideal route may be slightly faster than a reflected route, the time it takes to calculate that ideal route is generally longer than the time savings derived from this slightly better routing. dStor is therefore able to reflect requests very quickly and at a low system load, which serves as powerful protection against denial of service attacks on the system.
Because the dStor nodes operate as a self-healing mesh, (meaning each computer is networked to many other computers without any central server required, and can reconnect when links are broken) any end user that can connect to any node in the dStor network is effectively connected to all of the other nodes as well. This means that dStor is accessible from almost anywhere the internet operates. Even when governments block outside access, as long as one node behind that firewall can reach dStor, anyone who can connect to that node can connect to the whole dStor system.

Anyone can operate a dStor storage node if their infrastructure meets the minimum requirements. At present these minimums are:

  • a computer with an Intel i3 or better processor and 8GB of RAM running Unix (Ubuntu 18.4 LTS) with:
  • at least 4TB of hard drive space, and
  • 50Mbps of up bandwidth dedicated to dStor operation.
  • However, systems with more hard drive space can store (and be paid for) more data and those with lower latency or higher bandwidth can deliver (and be paid for) more data service. This incentivizes higher quality infrastructure for the good of the system.

Anyone can be a customer of dStor once we’re out of our Beta phase. Most of dStor’s direct customers will be app developers and most end users will access data stored on dStor via the apps that these developers create. The dStor interface is an API and therefore less user-friendly to non-developers, however anyone can create a dStor account if they choose and the API is easy to use and well documented.
dStor offers similar access times to S3 and other cloud services providers like Google Cloud and Microsoft Azure. However, dStor costs significantly less than these services (around 20-40% of the cost). dStor also delivers data from a decentralized system that does not have the single points of failure of centralized services. Also, because dStor stores data on many locations around the world, it can often provide faster access to data than requests from AWS in distant regions as well as better data permanence in the case of catastrophic events.
dStor and IPFS are both decentralized storage systems and can offer low storage costs. dStor shares some design elements with IPFS, but dStor is different in important ways. dStor’s gateway node system is much faster at serving data than IPFS. dStor also stores more copies of data than IPFS, which is designed to minimize data replication, whereas dStor is designed for a moderate and consistent amount of forward replication of data. This makes data much more permanent on dStor compared to IPFS, which can simply lose data from the system.
A filehash is the result of a hashing function applied to a file or data block that can be used as its uniform resource identifier (URI). dStor uses the SHA256 hashing algorithm to derive a URI for each block of data. This hashing function will create an entirely different filehash of the data block if even one byte is altered. Therefore, users can be assured that no files on dStor have been altered from their original versions. The SHA256 hashing function creates more possible results than there are atoms in the universe. The odds of users accessing a file without knowing the precise filehash are, therefore, infinitesimal.
The dStor system is currently in the first phase of its three-phase beta release. In this phase, the dStor storage node network is comprised of node operators with an existing relationship with dStor and dStor customers are a limited number of app developers. The goal of phase one of the beta release is to ensure that the interfaces and documentation are well prepared for the needs of more general users in phase two of the beta release. Phase one is expected to last approximately two months.
Documentation and tutorials are provided for using the dStor API at the dStor website and code repository.
Like other cloud storage services, storing data on dStor requires payment. Because maintaining stored data requires ongoing maintenance and operational costs, there is likely no economic system of paying for the eternal storage and fast delivery of data. Data will be stored on dStor only as long as the customer maintains payment for its ongoing storage and data delivery.
dStor does not use blockchain for storage functions, only for payment and account permission. Current blockchain technology is too slow to accommodate the high speed of data access required for modern app usage at scale. During the beta release, dStor will use the Telos blockchain as the first payment method. Other payment methods will be added to accommodate user needs.
dStor billing is structured similar to that of leading centralized cloud storage providers but with significantly lower pricing. There is a cost for the initial ingesting of data that is necessary to prevent abuse. There is also a monthly storage cost for maintaining data in the system and a cost for delivering data to end users. The pricing is intended to be more streamlined than centralized storage providers without tiers or fees for HTTP commands.
Region
Storage
PUT, COPY, POST, LIST
GET, SELECT
Data transfer in
Data transfer out 0-1GB
Data transfer out .001-10 TB
Data transfer out 10-50 TB
Data transfer out 50-150 TB
Data transfer out >150 TB

dStor vs. Amazon S3

dStor
Amazon S3
All region, hot storage
S3 Standard, 1 region storage
$0.015 Gib/month
35% savings
$0.0230 Gib/month
$0/1,000
100% savings
$0.0050/1,000
$0/1,000
100% savings
$0.0040/1,000
$0.015 Gib/month
63% savings
$0.0400 Gib/month
$0.050 Gib/month
$0.0000 Gib/month
$0.050 Gib/month
44% savings
$0.0900 Gib/month
$0.050 Gib/month
41% savings
$0.0850 Gib/month
$0.050 Gib/month
29% savings
$0.0700 Gib/month
$0.050 Gib/month
$0.0500 Gib/month
Region
Storage
PUT, COPY, POST, LIST
GET, SELECT
Data transfer in
Data transfer out 0-1GB
Data transfer out .001-10 TB
Data transfer out 10-50 TB
Data transfer out 50-150 TB
Data transfer out >150 TB

dStor vs. Microsoft Azure

dStor
Microsoft Azure
All region, hot storage
Block blob, locally redundant, 1 region storage
$0.015 Gib/month
28% savings
$0.0208 Gib/month
$0/1,000
100% savings
$0.0055/1,000
$0/1,000
100% savings
$0.0004/1,000
$0.015 Gib/month
$0.0000 Gib/month
$0.050 Gib/month
$0.0000 Gib/month
$0.050 Gib/month
42% savings
$0.0870 Gib/month
$0.050 Gib/month
40% savings
$0.0830 Gib/month
$0.050 Gib/month
29% savings
$0.0700 Gib/month
$0.050 Gib/month
$0.0500 Gib/month
Region
Storage
PUT, COPY, POST, LIST
GET, SELECT
Data transfer in
Data transfer out 0-1GB
Data transfer out .001-10 TB
Data transfer out 10-50 TB
Data transfer out 50-150 TB
Data transfer out >150 TB

dStor vs. Google Cloud

dStor
Google Cloud
All region, hot storage
Standard Storage
$0.015 Gib/month
42% savings
$0.0260 Gib/month
$0/1,000
100% savings
$0.0050/1,000
$0/1,000
100% savings
$0.0050/1,000
$0.015 Gib/month
$0.000 Gib/month
$0.050 Gib/month
58% savings
$0.1200 Gib/month
$0.050 Gib/month
55% savings
$0.1100 Gib/month
$0.050 Gib/month
38% savings
$0.0800 Gib/month
$0.050 Gib/month
38% savings
$0.0800 Gib/month
$0.050 Gib/month
38% savings
$0.0800 Gib/month

Customers establish an account with dStor and receive a customer number and designate their method of payment. Monthly invoices will be emailed to each customer for the previous month’s usage. Customers will have 15 days to pay their invoice in full or their stored assets will be flagged for deletion. (Customers may retain a positive payment balance to prevent files from being deleted.) When very high usage occurs, an invoice may be sent more often than monthly.

During the beta period, TLOS tokens on the Telos blockchain will be the first payment gateway. Additional forms of payment will be added in various fiat and crypto currencies as available.

To offer users the ability to evaluate dStor, account credits are available to users who request them and provide the necessary customer data. The account credit does not expire at a particular time, only when the full credit has been used.

Storage node operators share in the previous month’s income divided based on what percentage each provided of the total seed storage and data delivery. Approximately 25% of the total storage node operator payment amount is reserved for seed storage and 75% for data delivery storage.

The storage portion of the payment is divided among all storage nodes based on what percentage of the assigned data seeds are actually stored on their system. Each data file is seeded on approximately 16-19 individual nodes fairly randomly amongst adequately performing storage nodes based on capacity. Nodes with more storage capacity can earn a greater percentage of this pool. If a data seed is expected on a storage node but is found not to exist, the node is no longer paid for that particular seed and it is assigned to another node.

The data delivery portion of the payment is divided among all storage nodes based on what percentage of the data delivery that node performed. Nodes with more available up bandwidth dedicated to dStor and good system performance can expect to receive a higher proportion of the data delivery income.

Storage node operators are paid monthly roughly six weeks after the beginning of the billing cycle. dStor income is converted into TLOS by the system so that accounts may automatically pay and withdraw. Each provider will be paid directly into their Telos account by smart contract. In the future, we intend to have automated conversion into other fiat and crypto currencies.
dStor is capable of storing any type of data that users add to the system: clear, encrypted, or sharded. dStor does shard (split into multiple segments) data files over a certain size. Each shard of a file receives its own filehash and is stored on a different set of servers than other shards. dStor does not encrypt data directly. Encryption should be handled by the user or app interfacing with dStor. This is a protection for user data security. If data is sent to dStor unencrypted and flagged for dStor to encrypt it, then dStor would have access to this unencrypted data prior and could read and retain copies of the data. Users would need to trust dStor not to copy their data prior to encryption. A better approach is for users or apps to directly encrypt data prior to ingesting it into dStor.
Because each file is addressed via a filehash that is a hash of the file’s contents, it is impossible to access a file with changed data using its original name.
Once the dStor network exceeds 30 storage nodes, each new data seed file will be stored on 16 distinct nodes, expanding to about 19 nodes as the system expands. The dStor gateway nodes regularly poll storage nodes to confirm that the expected seed files exist where expected. Storage nodes are paid to maintain these seeds. If a seed is not maintained by a node as expected, dStor will assign the seed to a new node and no longer pay the original storage node for it. In this way, dStor maintains a high number of seed files and incentivizes storage nodes to retain files.
dStor storage nodes will be operated all over the world. We expect that many will be in professional data centers where operators already have a footprint and wish to earn additional income. Data centers themselves are likely to store dStor files as they can do so at high volume and performance with competitive costs. However, many home users with fiber-to-the-home service are likely to operate dStor storage nodes to make some income from their unused capacity. Most dStor files will be sharded into multiple file segments and stored on many different storage nodes so that files cannot be easily pieced together by anyone but their owners.
Yes, data can be deleted from dStor using the API or console app. When a customer deletes a file, a message is broadcast to all storage nodes to flag it for deletion. The files will then be removed in the next periodic garbage collection. However, dStor files are optimized for retention and forward replication to ensure that data is never lost from the system, so in some edge cases, files may persist after their scheduled deletion. When this occurs, issuing a new deletion command should eliminate the file in the next deletion cycle.
dStor can stream media using the HLS (HTTP Live Streaming) video standard with HTML 5 video players. These open standards provide high quality streaming that can dynamically adapt to varying internet connectivity performance. HLS breaks media files into ten-second sections that are streamed to the player. This maximizes the strengths of dStor. It should also enable blockchain based micropayments per media segment for dapps that wish to develop such services.

The storage and gateway operations of dStor are decentralized. Storage nodes are operated by companies and individuals around the world. Gateway nodes are operated by dStor and a small number of other trusted companies during the beta period. As dStor grows, more gateway nodes will be operated by additional independent companies, further decentralizing this component of dStor.

The only component of dStor that is not currently decentralized is the accounting system which must operate on a high-speed network in order to keep track of how users and storage node operators are using the system. This currently works faster than any decentralized blockchain can operate. (We helped launch two of the world’s fastest and highest capacity blockchains, EOS and Telos, so we have good insight into their operational speeds.) We expect that blockchains will grow fast enough handle to decentralizing this accounting function within a few years and our intention is to fully decentralize dStor as soon as it is feasible. The main beneficiaries of this decentralization will be the node operators who will no longer need to trust dStor to accurately account for their rewards. Customers already interact only with decentralized components of dStor. No storage or access element of dStor is centralized and a failure of the accounting system would only disable the ability to pay accounts for a brief time. The database structure of this is recoverable and physically decentralized but the servers are all operated by dStor, meaning that there is still an element of trust required in terms of accounting, but not in terms of data integrity.

Telos is the first blockchain used by dStor for controlling permission and collecting and distributing payments. Before long there will be multiple forms of payment available to customers including USD and other fiat currencies, as well as leading cryptocurrencies such as BTC, ETH, and EOS. dStor is likely to continue to pay storage node providers in Telos TLOS tokens but intends to add automated swap contracts to allow node operators to easily collect funds in a variety of currencies.

dStor is developed by GoodBlock Technologies, Inc., which is a company that was founded by staff from the Telos block producer and developer company known as GoodBlock. Many of the companies involved in building the initial dStor network are also affiliated with Telos as block producers and/or developers. Many of the first dapps being developed on the dStor platform are built on Telos. However, because dStor is decentralized storage, it can be used by projects that operate on any blockchain, or that don’t use a blockchain at all.