Provenance is the metadata that describes the history of objects. Provenance provides new functio... more Provenance is the metadata that describes the history of objects. Provenance provides new functionality in a variety of areas, including experimental documentation, debugging, search, and security. As a result, a number of groups have built systems to capture provenance. Most of these systems focus on provenance collection, a few systems focus on building applications that use the provenance, but all of these systems ignore an important aspect: efficient long-term storage of provenance. In this article, we first analyze the provenance collected from multiple workloads and characterize the properties of provenance with respect to long-term storage. We then propose a hybrid scheme that takes advantage of the graph structure of provenance data and the inherent duplication in provenance data. Our evaluation indicates that our hybrid scheme, a combination of Web graph compression (adapted for provenance) and dictionary encoding, provides the best trade-off in terms of compression ratio, ...
In this paper, we present the design and perfor mance evaluation of Oasis, an active storage fra... more In this paper, we present the design and perfor mance evaluation of Oasis, an active storage framework for object-based storage systems that complies with the current TI0 OSD standard. In contrast with previous work, Oasis has the following advantages. First, Oasis enables users to transparently process the OSD object and supports different process ing gran ularity (from the single object to all the objects in the OSD) by extending the OSD object attribute page defined in the TI0 OSD standard. Second, Oasis provides an easy and efficient way for users to manage the application functions in the OSD by using the existing OSD commands. Third, Oasis can authorize the execution of the application function in the OSD by enhancing the TI0 OSD security protocol, allowing only authorized users to use the system. We evaluate the performance and scalability of our system im plementation on Oasis by running three typical applications. The results indicate that active storage far outperforms the traditional object-based storage system in applications that filter data on the OSD. We also experiment with Java based applications and C based applications. Our experiments indicate that Java based applications may be bottlenecked for I/O-intensive applications, while for applications that do not heavily rely on the I/O op erations, both Java based applications and C based applications achieve comparable performance. Our microbenchmarks indicate that Oasis implementation overhead is minimal compared to the Intel OSD reference implementation, between 1.2% to 5.9% for Read commands and 0.6% to 9.9% for Write commands.
2010 Ninth International Symposium on Distributed Computing and Applications to Business, Engineering and Science, 2010
ABSTRACT With the development of storage technology, storage system capacity has been rapidly gro... more ABSTRACT With the development of storage technology, storage system capacity has been rapidly growing, so rapid that storage I/O performance has become one of the most critical issues in large complex systems. This article combines the latest research in object-based storage with replica technology to share I/O in the object storage system, and to maintain high and stable I/O response rate. Theoretical derivation and simulation experiment results are both examined, which show the prominent role that replicas can play in improving system performance and maintain stable I/O response rate.
ABSTRACT Provenance is a kind of metadata that identifies the origin or history of objects. Prove... more ABSTRACT Provenance is a kind of metadata that identifies the origin or history of objects. Provenance enables new functionality in a wide range of areas, including experimental documentation, security, search, debugging, etc. In this paper, we explore a new provenance application: provenance-based rebuild. Compared to the traditional ECC scheme, using provenance to reconstruct lost data has the salient advantage such as a more fine-grained reconstruction granularity and parallel rebuild. We present the experience in detailed designing and implementing this system, including a wide variety of issues that we have to solve. We also propose to utilize some classical techniques such as active storage technology to accelerate the provenance-based rebuild performance.
Provenance is a kind of metadata that identifies the origin or history of objects. Provenance ena... more Provenance is a kind of metadata that identifies the origin or history of objects. Provenance enables new functionality in a wide range of areas, including experimental documentation, security, search, debugging, etc. In this paper, we explore a new provenance application: provenance-based rebuild. Compared to the traditional ECC scheme, using provenance to reconstruct lost data has the salient advantage such as a more fine-grained reconstruction granularity and parallel rebuild. We present the experience in detailed designing and implementing this system, including a wide variety of issues that we have to solve. We also propose to utilize some classical techniques such as active storage technology to accelerate the provenance-based rebuild performance.
Proceedings of the 21st ACM international conference on Information and knowledge management - CIKM '12, 2012
ABSTRACT Efficient provenance storage is an essential step towards the adoption of provenance. In... more ABSTRACT Efficient provenance storage is an essential step towards the adoption of provenance. In this paper, we analyze the provenance collected from multiple workloads with a view towards efficient storage. Based on our analysis, we characterize the properties of provenance with respect to long term storage. We then propose a hybrid scheme that takes advantage of the graph structure of provenance data and the inherent duplication in provenance data. Our evaluation indicates that our hybrid scheme, a combination of web graph compression (adapted for provenance) and dictionary encoding, provides the best tradeoff in terms of compression ratio, compression time and query performance when compared to other compression schemes.
2011 IEEE 27th Symposium on Mass Storage Systems and Technologies (MSST), 2011
ABSTRACT In this paper, we present the design and performance evaluation of Oasis, an active stor... more ABSTRACT In this paper, we present the design and performance evaluation of Oasis, an active storage framework for object-based storage systems that complies with the current T10 OSD standard. In contrast with previous work, Oasis has the following advantages. First, Oasis enables users to transparently process the OSD object and supports different processing granularity (from the single object to all the objects in the OSD) by extending the OSD object attribute page defined in the T10 OSD standard. Second, Oasis provides an easy and efficient way for users to manage the application functions in the OSD by using the existing OSD commands. Third, Oasis can authorize the execution of the application function in the OSD by enhancing the T10 OSD security protocol, allowing only authorized users to use the system. We evaluate the performance and scalability of our system implementation on Oasis by running three typical applications. The results indicate that active storage far outperforms the traditional object-based storage system in applications that filter data on the OSD. We also experiment with Java based applications and C based applications. Our experiments indicate that Java based applications may be bottlenecked for I/O-intensive applications, while for applications that do not heavily rely on the I/O operations, both Java based applications and C based applications achieve comparable performance. Our microbenchmarks indicate that Oasis implementation overhead is minimal compared to the Intel OSD reference implementation, between 1.2% to 5.9% for Read commands and 0.6% to 9.9% for Write commands.
ABSTRACT Previous storage reliability solutions mainly include log files, snapshot, backup or ECC... more ABSTRACT Previous storage reliability solutions mainly include log files, snapshot, backup or ECC-based schemes. These technologies focus more on the whole disk or system reliability, but not single data object. However, it is not efficient to rebuild a file by recovering a whole system using the technologies above. In this paper, we propose to use provenance, the origin or history of objects, to rebuild damaged or lost files. Provenance-based rebuild can exactly reconstruct the right file lost or damaged by backtracking its generation process in the past. The evaluation results show that provenance-based rebuild performs significantly better than log-based technology with reasonable time and space overhead.
The provenance community has built a number of systems to collect provenance, most of which assum... more The provenance community has built a number of systems to collect provenance, most of which assume that provenance will be retained indefinitely. However, it is not cost-effective to retain provenance information inefficiently. Since provenance can be viewed as a graph, we note the similarities to web graphs and draw upon techniques from the web compression domain to provide our own novel and improved graph compression solutions for provenance graphs. Our preliminary results show that adapting web compression techniques results in a compression ratio of 2.12:1 to 2.71:1, which we can improve upon to reach ratios of up to 3.31:1.
Provenance is the metadata that describes the history of objects. Provenance provides new functio... more Provenance is the metadata that describes the history of objects. Provenance provides new functionality in a variety of areas, including experimental documentation, debugging, search, and security. As a result, a number of groups have built systems to capture provenance. Most of these systems focus on provenance collection, a few systems focus on building applications that use the provenance, but all of these systems ignore an important aspect: efficient long-term storage of provenance. In this article, we first analyze the provenance collected from multiple workloads and characterize the properties of provenance with respect to long-term storage. We then propose a hybrid scheme that takes advantage of the graph structure of provenance data and the inherent duplication in provenance data. Our evaluation indicates that our hybrid scheme, a combination of Web graph compression (adapted for provenance) and dictionary encoding, provides the best trade-off in terms of compression ratio, compression time, and query performance when compared to other compression schemes.
• We provide transparent and multi-granularity processing for function execution. • We provide di... more • We provide transparent and multi-granularity processing for function execution. • We provide digital signature scheme for ensuring the code integrity. • We can integrate the Oasis framework into the Kinetic cloud platform. • We implement and evaluate the adaptive computation partition method in Oasis. a b s t r a c t The network bottleneck incurred by big data process and transfer has increasingly become a severe problem in today's data center and cloud. Exploring and exploiting the advantages of both the scalable object storage architecture and intelligent active storage technology are one of the ways to address this challenge. In this paper, we present the design and performance evaluation of Oasis, an active storage framework for object-based storage platform such as Seagate Kinetic. The basic idea behind Oasis is to leverage the OSD's processing capability to run data intensive applications locally. In contrast with previous work, Oasis has the following advantages. First, Oasis enables users to transparently process the OSD object and supports different processing granularity. Second, Oasis can ensure the integrity of execution code using signature scheme and provide the access control for the code execution in the OSD by enhancing the existing OSD security protocol. Third, Oasis can partition the computation task between host and OSD dynamically according to the OSD workload status. Our work on Oasis can be integrated into Kinetic object storage platform seamlessly. Experimental results on widely-used real world applications demonstrate the performance and efficiency of our system.
Provenance is the metadata that describes the history of objects. Provenance provides new functio... more Provenance is the metadata that describes the history of objects. Provenance provides new functionality in a variety of areas, including experimental documentation, debugging, search, and security. As a result, a number of groups have built systems to capture provenance. Most of these systems focus on provenance collection, a few systems focus on building applications that use the provenance, but all of these systems ignore an important aspect: efficient long-term storage of provenance. In this article, we first analyze the provenance collected from multiple workloads and characterize the properties of provenance with respect to long-term storage. We then propose a hybrid scheme that takes advantage of the graph structure of provenance data and the inherent duplication in provenance data. Our evaluation indicates that our hybrid scheme, a combination of Web graph compression (adapted for provenance) and dictionary encoding, provides the best trade-off in terms of compression ratio, ...
In this paper, we present the design and perfor mance evaluation of Oasis, an active storage fra... more In this paper, we present the design and perfor mance evaluation of Oasis, an active storage framework for object-based storage systems that complies with the current TI0 OSD standard. In contrast with previous work, Oasis has the following advantages. First, Oasis enables users to transparently process the OSD object and supports different process ing gran ularity (from the single object to all the objects in the OSD) by extending the OSD object attribute page defined in the TI0 OSD standard. Second, Oasis provides an easy and efficient way for users to manage the application functions in the OSD by using the existing OSD commands. Third, Oasis can authorize the execution of the application function in the OSD by enhancing the TI0 OSD security protocol, allowing only authorized users to use the system. We evaluate the performance and scalability of our system im plementation on Oasis by running three typical applications. The results indicate that active storage far outperforms the traditional object-based storage system in applications that filter data on the OSD. We also experiment with Java based applications and C based applications. Our experiments indicate that Java based applications may be bottlenecked for I/O-intensive applications, while for applications that do not heavily rely on the I/O op erations, both Java based applications and C based applications achieve comparable performance. Our microbenchmarks indicate that Oasis implementation overhead is minimal compared to the Intel OSD reference implementation, between 1.2% to 5.9% for Read commands and 0.6% to 9.9% for Write commands.
2010 Ninth International Symposium on Distributed Computing and Applications to Business, Engineering and Science, 2010
ABSTRACT With the development of storage technology, storage system capacity has been rapidly gro... more ABSTRACT With the development of storage technology, storage system capacity has been rapidly growing, so rapid that storage I/O performance has become one of the most critical issues in large complex systems. This article combines the latest research in object-based storage with replica technology to share I/O in the object storage system, and to maintain high and stable I/O response rate. Theoretical derivation and simulation experiment results are both examined, which show the prominent role that replicas can play in improving system performance and maintain stable I/O response rate.
ABSTRACT Provenance is a kind of metadata that identifies the origin or history of objects. Prove... more ABSTRACT Provenance is a kind of metadata that identifies the origin or history of objects. Provenance enables new functionality in a wide range of areas, including experimental documentation, security, search, debugging, etc. In this paper, we explore a new provenance application: provenance-based rebuild. Compared to the traditional ECC scheme, using provenance to reconstruct lost data has the salient advantage such as a more fine-grained reconstruction granularity and parallel rebuild. We present the experience in detailed designing and implementing this system, including a wide variety of issues that we have to solve. We also propose to utilize some classical techniques such as active storage technology to accelerate the provenance-based rebuild performance.
Provenance is a kind of metadata that identifies the origin or history of objects. Provenance ena... more Provenance is a kind of metadata that identifies the origin or history of objects. Provenance enables new functionality in a wide range of areas, including experimental documentation, security, search, debugging, etc. In this paper, we explore a new provenance application: provenance-based rebuild. Compared to the traditional ECC scheme, using provenance to reconstruct lost data has the salient advantage such as a more fine-grained reconstruction granularity and parallel rebuild. We present the experience in detailed designing and implementing this system, including a wide variety of issues that we have to solve. We also propose to utilize some classical techniques such as active storage technology to accelerate the provenance-based rebuild performance.
Proceedings of the 21st ACM international conference on Information and knowledge management - CIKM '12, 2012
ABSTRACT Efficient provenance storage is an essential step towards the adoption of provenance. In... more ABSTRACT Efficient provenance storage is an essential step towards the adoption of provenance. In this paper, we analyze the provenance collected from multiple workloads with a view towards efficient storage. Based on our analysis, we characterize the properties of provenance with respect to long term storage. We then propose a hybrid scheme that takes advantage of the graph structure of provenance data and the inherent duplication in provenance data. Our evaluation indicates that our hybrid scheme, a combination of web graph compression (adapted for provenance) and dictionary encoding, provides the best tradeoff in terms of compression ratio, compression time and query performance when compared to other compression schemes.
2011 IEEE 27th Symposium on Mass Storage Systems and Technologies (MSST), 2011
ABSTRACT In this paper, we present the design and performance evaluation of Oasis, an active stor... more ABSTRACT In this paper, we present the design and performance evaluation of Oasis, an active storage framework for object-based storage systems that complies with the current T10 OSD standard. In contrast with previous work, Oasis has the following advantages. First, Oasis enables users to transparently process the OSD object and supports different processing granularity (from the single object to all the objects in the OSD) by extending the OSD object attribute page defined in the T10 OSD standard. Second, Oasis provides an easy and efficient way for users to manage the application functions in the OSD by using the existing OSD commands. Third, Oasis can authorize the execution of the application function in the OSD by enhancing the T10 OSD security protocol, allowing only authorized users to use the system. We evaluate the performance and scalability of our system implementation on Oasis by running three typical applications. The results indicate that active storage far outperforms the traditional object-based storage system in applications that filter data on the OSD. We also experiment with Java based applications and C based applications. Our experiments indicate that Java based applications may be bottlenecked for I/O-intensive applications, while for applications that do not heavily rely on the I/O operations, both Java based applications and C based applications achieve comparable performance. Our microbenchmarks indicate that Oasis implementation overhead is minimal compared to the Intel OSD reference implementation, between 1.2% to 5.9% for Read commands and 0.6% to 9.9% for Write commands.
ABSTRACT Previous storage reliability solutions mainly include log files, snapshot, backup or ECC... more ABSTRACT Previous storage reliability solutions mainly include log files, snapshot, backup or ECC-based schemes. These technologies focus more on the whole disk or system reliability, but not single data object. However, it is not efficient to rebuild a file by recovering a whole system using the technologies above. In this paper, we propose to use provenance, the origin or history of objects, to rebuild damaged or lost files. Provenance-based rebuild can exactly reconstruct the right file lost or damaged by backtracking its generation process in the past. The evaluation results show that provenance-based rebuild performs significantly better than log-based technology with reasonable time and space overhead.
The provenance community has built a number of systems to collect provenance, most of which assum... more The provenance community has built a number of systems to collect provenance, most of which assume that provenance will be retained indefinitely. However, it is not cost-effective to retain provenance information inefficiently. Since provenance can be viewed as a graph, we note the similarities to web graphs and draw upon techniques from the web compression domain to provide our own novel and improved graph compression solutions for provenance graphs. Our preliminary results show that adapting web compression techniques results in a compression ratio of 2.12:1 to 2.71:1, which we can improve upon to reach ratios of up to 3.31:1.
Provenance is the metadata that describes the history of objects. Provenance provides new functio... more Provenance is the metadata that describes the history of objects. Provenance provides new functionality in a variety of areas, including experimental documentation, debugging, search, and security. As a result, a number of groups have built systems to capture provenance. Most of these systems focus on provenance collection, a few systems focus on building applications that use the provenance, but all of these systems ignore an important aspect: efficient long-term storage of provenance. In this article, we first analyze the provenance collected from multiple workloads and characterize the properties of provenance with respect to long-term storage. We then propose a hybrid scheme that takes advantage of the graph structure of provenance data and the inherent duplication in provenance data. Our evaluation indicates that our hybrid scheme, a combination of Web graph compression (adapted for provenance) and dictionary encoding, provides the best trade-off in terms of compression ratio, compression time, and query performance when compared to other compression schemes.
• We provide transparent and multi-granularity processing for function execution. • We provide di... more • We provide transparent and multi-granularity processing for function execution. • We provide digital signature scheme for ensuring the code integrity. • We can integrate the Oasis framework into the Kinetic cloud platform. • We implement and evaluate the adaptive computation partition method in Oasis. a b s t r a c t The network bottleneck incurred by big data process and transfer has increasingly become a severe problem in today's data center and cloud. Exploring and exploiting the advantages of both the scalable object storage architecture and intelligent active storage technology are one of the ways to address this challenge. In this paper, we present the design and performance evaluation of Oasis, an active storage framework for object-based storage platform such as Seagate Kinetic. The basic idea behind Oasis is to leverage the OSD's processing capability to run data intensive applications locally. In contrast with previous work, Oasis has the following advantages. First, Oasis enables users to transparently process the OSD object and supports different processing granularity. Second, Oasis can ensure the integrity of execution code using signature scheme and provide the access control for the code execution in the OSD by enhancing the existing OSD security protocol. Third, Oasis can partition the computation task between host and OSD dynamically according to the OSD workload status. Our work on Oasis can be integrated into Kinetic object storage platform seamlessly. Experimental results on widely-used real world applications demonstrate the performance and efficiency of our system.
Uploads