Apache HDFS (Hadoop Distributed File System) es un sistema de archivos diseñado para almacenar archivos de gran tamaño, distribuidos bloque por bloque entre los nodos de un clúster informático.
Apache Hive es un sistema de gestión de bases de datos basado en la plataforma Hadoop.
Apache HBase es un proyecto DBMS NoSQL de código abierto del ecosistema Hadoop.
Apache KNOX, una API REST y puerta de enlace de aplicaciones para los componentes del ecosistema Apache Hadoop, proporciona un único punto de acceso para todas las conexiones HTTP a los clústeres de Apache Hadoop y un sistema de inicio de sesión único (SSO) para los servicios y la interfaz de usuario de los componentes de Apache Hadoop.
Apache Ranger es un marco para proporcionar, monitorear y administrar la seguridad de datos de un extremo a otro en la plataforma Hadoop.
Traducción de una publicación de 2015 de Securing hdfs hive hbase knox ranger . No pude encontrar una publicación mejor y más nueva.
Introducción
Apache Ranger proporciona un enfoque integral para proteger un clúster de Hadoop. Proporciona una administración centralizada de políticas de seguridad para los requisitos clave de seguridad empresarial, incluida la autorización, la contabilidad y la protección de datos.
Apache Ranger amplía la funcionalidad básica para un uso coordinado en las cargas de trabajo de Hadoop, incluido el SQL interactivo por lotes y Hadoop en tiempo real.
En este tutorial, veremos el uso de Apache Ranger para HDP 2.3 para proteger su entorno Hadoop. Cubriremos los siguientes temas:
Admite la autorización y auditoría de Knox
Políticas de línea de comandos en Hive
Políticas de línea de comando en HBase
API REST para Policy Manager
Prerrequisitos
Hortonworks Sandbox, SSH:
Knox LDAP
Ambari http://localhost:8080/ ( — admin admin ) Knox .
Service Actions
Start
.
Knox :
Service Actions
Knox
Start Demo LDAP
.
- LDAP :
Knox
, Ranger http://localhost:6080/ -. — admin
, — admin
.
,
sudo service ranger-admin start
sandbox_knox Knox Ranger Administration Portal
, .
Knox, global knox allow
.
Sandbox for Guest
Ranger Admin .
Sandbox for Guest
SSHd ( Sandbox) CURL WebHDFS.
curl -k -u admin:admin-password 'https://127.0.0.1:8443/gateway/knox_sample/webhdfs/v1?op=LISTSTATUS'
Ranger Policy Manager → Audit screen , () Knox.
CURL, guest
.
curl -k -u guest:guest-password 'https://127.0.0.1:8443/gateway/knox_sample/webhdfs/v1?op=LISTSTATUS'
{"FileStatuses":{"FileStatus":[{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16393,"group":"hadoop","length":0,"modificationTime":1439987528048,"owner":"yarn","pathSuffix":"app-logs","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":4,"fileId":16389,"group":"hdfs","length":0,"modificationTime":1439987809562,"owner":"hdfs","pathSuffix":"apps","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":17000,"group":"hdfs","length":0,"modificationTime":1439989173392,"owner":"hdfs","pathSuffix":"demo","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16398,"group":"hdfs","length":0,"modificationTime":1439987529660,"owner":"hdfs","pathSuffix":"hdp","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16394,"group":"hdfs","length":0,"modificationTime":1439987528532,"owner":"mapred","pathSuffix":"mapred","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":2,"fileId":16396,"group":"hadoop","length":0,"modificationTime":1439987538099,"owner":"mapred","pathSuffix":"mr-history","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16954,"group":"hdfs","length":0,"modificationTime":1439988741413,"owner":"hdfs","pathSuffix":"ranger","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":3,"fileId":16386,"group":"hdfs","length":0,"modificationTime":1440165443820,"owner":"hdfs","pathSuffix":"tmp","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":8,"fileId":16387,"group":"hdfs","length":0,"modificationTime":1439988397561,"owner":"hdfs","pathSuffix":"user","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}]}}
Ranger Policy Manager → Audit screen.
Ranger Knox , Knox, , Ranger.
Knox Ranger, (WebHDFS, WebHCAT . .) , / IP-
grant/revoke Hive
Ranger grant
/revoke
, Hue for Hive
. Ranger , , Hive .
Hive Ranger.
Grant
, Hive
. beeline,
beeline -u "jdbc:hive2://sandbox.hortonworks.com:10000/default" -n it1 -p it1-d org.apache.hive.jdbc.HiveDriver
GRANT
grant select, update on table xademo.customer_details to user network1;
:
Ranger Administration Portal → Audit
, it1
.
Ranger it1
. Ranger , .
beeline , .
GRANT select, update on table xademo.customer_details to user network1;
, , / Ranger Admin Portal → Policy Manager. , , .
?
Ranger GRANT/REVOKE
Hive
. Hive
(Hiveserver2
).
GRANT
REVOKE
.
grant/revoke HBase
Ranger grant/revoke
, Hbase
. Hive, Ranger Hbase .
, , HBase Ambari — http://127.0.0.1:8080 ( admin
).
, Service Actions
.
Grant Hbase.
HBase Global Allow
Ranger Administration Portal — .
HBase it1
su - it1 [it1@sandbox ~]$ hbase shell
grant, mktg1
, iemployee
.
hbase(main):001:0> grant 'mktg1', 'RWC', 'iemployee'
, :
Ranger Administration Portal→ Policy Manager , admin
it1
.
HBase.
hbase(main):006:0> grant 'mktg1', 'RWC', 'iemployee' 0 row(s) in 0.8670 seconds
HBase Ranger Policy Administration. iemployee
, .
, Ranger. HBase
hbase(main):007:0> revoke 'mktg1', 'iemployee' 0 row(s) in 0.4330 seconds
,
?
Ranger GRANT/REVOKE
Hbase . .
GRANT
REVOKE .
REST APIs
Ranger REST API. API , , .
REST API
CURL. API hadoopdev-testing-policy2
HDFS sandbox_hdfs
curl -i --header "Accept:application/json" -H "Content-Type: application/json" --user admin:admin -X POST http://127.0.0.1:6080/service/public/api/policy -d '{ "policyName":"hadoopdev-testing-policy2","resourceName":"/demo/data/test","description":"Testing policy for /demo/data/test","repositoryName":"sandbox_hdfs","repositoryType":"HDFS","permMapList":[{"userList":["mktg1"],"permList":["Read"]},{"groupList":["IT"],"permList":["Read"]}],"isEnabled":true,"isRecursive":true,"isAuditEnabled":true,"version":"0.1.0","replacePerm":false}'
hadoopdev-testing-policy2
.
URL- http://127.0.0.1:6080/index.html#!/hdfs/1/policy/26
.
CURL, API.
curl -i --user admin:admin -X GET http://127.0.0.1:6080/service/public/api/policy/26
?
REST API. API , REST API Ranger.
Con suerte, en este vertiginoso recorrido por Ranger, habrá experimentado la simplicidad y el poder de Ranger para la gestión de la seguridad.