Data security in Hadoop will always be of upmost importance. Ever since there has been an increase in the use of Hadoop, it also entails the deployment of more security mechanisms. There is a high demand to encrypt data to ensure of its secrecy and that no privacy rule has been violated. But to ensure the security of both data at rest and in-transit, encryption methods must be implemented. Blocks of data along with their replicas are stored on different Datanodes by the HDFS to provide fault tolerance depending on the replication factor, which takes an enormous amount of storage. There is usually two replicas of the original block, each replica uses 100% overhead storage, resulting in 200% overhead storage. These replicas also consume resources, such as bandwidth usage. Therefore Erasure Coding (EC) is use in HDFS to provide the same level of fault tolerance and data durability and as well reduce the storage overhead by approximately 50%. Hadoop Cluster applies the transparent encryption method as a solution for data security. And HDFS Erasure Coding allow data accessibility at a reduced capacity. With EC, big data is stored in granular form and redundancy only requires 33% overhead. In this work, we evaluate the performance of EC when encryption is also applied. On top of the default EC policies (3–2, 6–3, 10–4), five new policies are also evaluated. In terms of security mechanisms, AES, 3DES and RC4 are evaluated.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Performance Evaluation of Security Mechanisms for Erasure Coded System in Hadoop

  • Aatish Chiniah

摘要

Data security in Hadoop will always be of upmost importance. Ever since there has been an increase in the use of Hadoop, it also entails the deployment of more security mechanisms. There is a high demand to encrypt data to ensure of its secrecy and that no privacy rule has been violated. But to ensure the security of both data at rest and in-transit, encryption methods must be implemented. Blocks of data along with their replicas are stored on different Datanodes by the HDFS to provide fault tolerance depending on the replication factor, which takes an enormous amount of storage. There is usually two replicas of the original block, each replica uses 100% overhead storage, resulting in 200% overhead storage. These replicas also consume resources, such as bandwidth usage. Therefore Erasure Coding (EC) is use in HDFS to provide the same level of fault tolerance and data durability and as well reduce the storage overhead by approximately 50%. Hadoop Cluster applies the transparent encryption method as a solution for data security. And HDFS Erasure Coding allow data accessibility at a reduced capacity. With EC, big data is stored in granular form and redundancy only requires 33% overhead. In this work, we evaluate the performance of EC when encryption is also applied. On top of the default EC policies (3–2, 6–3, 10–4), five new policies are also evaluated. In terms of security mechanisms, AES, 3DES and RC4 are evaluated.