Introducción
Déjame explicarte mi acordeón de inmediato. Sí, Internet está lleno de manuales. Sí, lleno de tutoriales por turnos. Sí, se puede decir que todo se mastica masticado. Pero específicamente en mi caso, como siempre sucede, hubo un puñado de "peros":
Hay manuales sobre cómo configurar un paquete de registro NiFi y NiFi con autenticación y autorización habilitadas. Pero ... se utilizan certificados autofirmados.
Hay manuales separados sobre cómo atornillar un certificado comercial para NiFi; respectivamente para el Registro NiFi "kagbe es el mismo". Pero la autenticación y autorización mutuas ocurrirán con el uso de SSL bidireccional ... y tenemos LDAP ... y entonces no podrá garantizar la coherencia de la dulce pareja utilizando solo el directorio externo en su intuición pura.
Hay manuales para conectarse con LDAP tanto para NiFi como para NiFi Registry. Nooo ... como en el anterior "pero", surgen preguntas, cómo arreglárselas con sólo LDAP, porque todavía tenemos NiFi Cli, pero él no sabe cómo hacer LDAP.
En otras palabras, todos los manuales tienen un pequeño matiz: cubren solo los escenarios más simples. Simplemente no hay documentación para paquetes complejos. Además, en el curso de la creación del paquete, encontré dificultades que se encuentran solo unas pocas veces en la red burguesa y todas carecen de respuestas o las respuestas no son relevantes.
Que tenemos y que queremos conseguir
Tenemos:
Certificado SSL / TLS comodín comercial;
Un servidor de directorio que puede hacer LDAP (en mi caso, Apache DS);
El servidor NiFi en el que se lleva a cabo el desarrollo;
Servidor NiFi en un entorno de comestibles;
Servidor de registro NiFi.
Queremos:
Autenticación, autorización y HTTPS. Además, NiFi no le permite habilitar mecanismos de autenticación integrados sin HTTPS;
Gestión de usuarios centralizada en un servidor de directorio;
Control de versiones del flujo y del grupo de procesos de una manera similar a gith;
.
:
Apache DS;
Java Keystore Java Truststore, HTTPS;
LDAP NiFi;
LDAP NiFi Registry;
;
NiFi NiFi Registry;
NiFi NiFi Cli.
wildcard : *.love.you;
FQDN Apache DS: auth.love.you;
Apache DS: uid=admin,ou=system; : secret;
Apache DS love.you: dc=love,dc=you;
Apache DS nifiAdmin nifiUser NiFi NiFi Registry;
Apache DS nifiAdmins nifiUsers NiFi NiFi Registry;
FQDN NiFi, : nifi-deve.love.you;
FQDN NiFi Registry: nifi-re.love.you;
FQDN () NiFi: nifi.love.you;
, NiFi: $NIFI_HOME;
, NiFi Registry: $NIFI_REGISTRY_HOME;
, NiFi Toolkit: $NIFI_TOOLKIT_HOME;
, : /love/conf/cert;
PKCS12, Java Keystore Java Truststore:
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
Apache DS
, , Apache DS . . Azure AD, " ", NiFi, AD' . Apache DS , , Apache.
10389 . Apache Directory Studio. , , .
New connection:
:
: uid=admin,ou=system secret. Check Authentication:
.
. - member objectclass inetOrgPerson. LDAP Browser' ou=schema Reload Entry:
cn=inetorgperson ou=objectClasses. m-oid=2.16.840.1.113730.3.2.2. Open With -> LDIF Entry Editor:
m-may: member Ctrl+S:
Open Configuration:
Advanced Partition configuration...
Add, ID Suffix. Ctrl+S:
: . . . :
. - LDIF . - import.ldif .
import.ldif
dn: ou=people,dc=love,dc=you
objectclass: organizationalUnit
objectClass: extensibleObject
objectclass: top
ou: people
dn: ou=groups,dc=love,dc=you
objectclass: organizationalUnit
objectClass: extensibleObject
objectclass: top
ou: groups
dn: cn=nifiUsers,ou=groups,dc=love,dc=you
objectClass: groupOfUniqueNames
objectClass: top
cn: nifiUsers
uniqueMember: cn=nifiUser,ou=people,dc=love,dc=you
dn: cn=nifiAdmins,ou=groups,dc=love,dc=you
objectClass: groupOfUniqueNames
objectClass: top
cn: nifiAdmins
uniqueMember: cn=nifiAdmin,ou=people,dc=love,dc=you
dn: cn=nifiUser,ou=people,dc=love,dc=you
objectclass: inetOrgPerson
objectclass: organizationalPerson
objectclass: person
objectclass: top
cn: nifiUser
description: A nifiUser user
sn: nifiUser
uid: nifiUser
mail: nifiUser@love.you
userpassword: password
member: cn=nifiUsers,ou=groups,dc=love,dc=you
dn: cn=nifiAdmin,ou=people,dc=love,dc=you
objectclass: inetOrgPerson
objectclass: organizationalPerson
objectclass: person
objectclass: top
cn: nifiAdmin
description: A nifiAdmin user
sn: nifiAdmin
uid: nifiAdmin
mail: nifiAdmin@love.you
userpassword: password
member: cn=nifiAdmins,ou=groups,dc=love,dc=you
dn: cn=*.love.you,ou=people,dc=love,dc=you
objectClass: person
objectClass: top
cn: *.love.you
sn: *.love.you
LDIF Import:
:
:
cn=*.love.you
, ou=people cn=*.love.you, objectClass: person:
, : , *.love.you. NiFi Registry.
Java Keystore Java Truststore
:
PEM-: fullchain.crt;
PEM-: private.key;
certificate authority (CA) PEM-: cacert.pem;
, . , .
. :
-----BEGIN CERTIFICATE----- ... ... ... -----END CERTIFICATE-----
.
, .
, , , ( ) . CA .
Java Keystore . PKCS12:
openssl pkcs12 -export -out nifi.p12 -inkey private.key -in fullchain.crt -name nifi-key
PKCS12 . .
Java Keystore PKCS12 :
keytool -importkeystore -srckeystore nifi.p12 -srcstoretype pkcs12 -srcalias nifi-key -destkeystore keystore.jks -deststoretype jks -destalias nifi-key
, . Java Keystore, .
Java Truststore:
keytool -importcert -alias nifi-cert -file cacert.pem -keystore truststore.jks
Java Truststore. . Keystore Truststore. . .
JKS NiFi, , NiFi NiFi Registry, /love/conf/cert/;
NiFi , , $NIFI_HOME/config/nifi.properties, .
nifi.properties
# Site to Site properties
nifi.remote.input.host=nifi-deve.love.you
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10443
# web properties #
nifi.web.http.host=
nifi.web.http.port=
nifi.web.https.host=nifi-deve.love.you
nifi.web.https.port=9443
# security properties #
nifi.security.keystore=/love/conf/cert/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.truststore=/love/conf/cert/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=ldap-provider
: nifi.web.http.host nifi.web.http.port. , , NiFi .
21 - , .
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
NiFi , $NIFI_HOME/config/nifi.properties, .
nifi.properties
# Site to Site properties
nifi.remote.input.host=nifi.love.you
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10443
# web properties #
nifi.web.http.host=
nifi.web.http.port=
nifi.web.https.host=nifi.love.you
nifi.web.https.port=9443
# security properties #
nifi.security.keystore=/love/conf/cert/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.truststore=/love/conf/cert/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=ldap-provider
# Identity Mapping Properties #
nifi.security.identity.mapping.pattern.dn=^CN=(.*?)$
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=NONE
: . .
21 - , .
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
NiFi Registry, $NIFI_REGISTRY_HOME/config/nifi-registry.properties, .
nifi-registry.properties
# web properties #
nifi.registry.web.http.host=
nifi.registry.web.http.port=
nifi.registry.web.https.host=nifi-re.love.you
nifi.registry.web.https.port=8443
# security properties #
nifi.registry.security.keystore=/abc/conf/cert/keystore.jks
nifi.registry.security.keystoreType=jks
nifi.registry.security.keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.registry.security.truststore=/abc/conf/cert/truststore.jks
nifi.registry.security.truststoreType=jks
nifi.registry.security.truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.registry.security.needClientAuth=false
nifi.registry.security.identity.provider=ldap-identity-provider
# Identity Mapping Properties #
nifi.registry.security.identity.mapping.pattern.dn=^CN=(.*?)$
nifi.registry.security.identity.mapping.value.dn=$1
nifi.registry.security.identity.mapping.transform.dn=NONE
: . .
16 - , .
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
LDAP NiFi
LDAP : $NIFI_HOME/conf/login-identity-providers.xml $NIFI_HOME/conf/authorizers.xml
login-identity-providers.xml . LDAP- . .
login-identity-providers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<loginIdentityProviders>
<provider>
<identifier>ldap-provider</identifier>
<class>org.apache.nifi.ldap.LdapProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">uid=admin,ou=system</property>
<property name="Manager Password">secret</property>
<property name="TLS - Keystore"></property>
<property name="TLS - Keystore Password"></property>
<property name="TLS - Keystore Type"></property>
<property name="TLS - Truststore"></property>
<property name="TLS - Truststore Password"></property>
<property name="TLS - Truststore Type"></property>
<property name="TLS - Client Auth"></property>
<property name="TLS - Protocol"></property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://auth.love.you:10389</property>
<property name="User Search Base">ou=people,dc=love,dc=you</property>
<property name="User Search Filter">(uid={0})</property>
<property name="Identity Strategy">USE_USERNAME</property>
<property name="Authentication Expiration">12 hours</property>
</provider>
</loginIdentityProviders>
, 27 29. 27 , , LDAP-. {0} - , User NiFi; uid - , , . , nifiUser.
, . ApacheDS.
29 - : (USE_USERNAME), DN (USE_DN). , , . DN .
authorizers.xml , NiFi "" . .
authorizers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizers>
<userGroupProvider>
<identifier>ldap-user-group-provider</identifier>
<class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">uid=admin,ou=system</property>
<property name="Manager Password">secret</property>
<property name="TLS - Keystore"></property>
<property name="TLS - Keystore Password"></property>
<property name="TLS - Keystore Type"></property>
<property name="TLS - Truststore"></property>
<property name="TLS - Truststore Password"></property>
<property name="TLS - Truststore Type"></property>
<property name="TLS - Client Auth"></property>
<property name="TLS - Protocol"></property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://auth.love.you:10389</property>
<property name="Page Size"></property>
<property name="Sync Interval">1 mins</property>
<property name="Group Membership - Enforce Case Sensitivity">false</property>
<property name="User Search Base">ou=people,dc=love,dc=you</property>
<property name="User Object Class">person</property>
<property name="User Search Scope">ONE_LEVEL</property>
<property name="User Search Filter"></property>
<property name="User Identity Attribute">cn</property>
<property name="User Group Name Attribute">member</property>
<property name="User Group Name Attribute - Referenced Group Attribute"></property>
<property name="Group Search Base">ou=groups,dc=love,dc=you</property>
<property name="Group Object Class">groupOfUniqueNames</property>
<property name="Group Search Scope">ONE_LEVEL</property>
<property name="Group Search Filter"></property>
<property name="Group Name Attribute">cn</property>
<property name="Group Member Attribute"></property>
<property name="Group Member Attribute - Referenced User Attribute"></property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">ldap-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">nifiAdmin</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1"></property>
<property name="Node Group"></property>
</accessPolicyProvider>
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers>
:
31 - objectClass ApacheDS, ;
35 - , ;
39 - objectClass, ;
50 - , ( identifier: <identifier>ldap-user-group-provider</identifier> <userGroupProvider>);
52 - , - "" .
NiFi nifiAdmin , ldif-.
LDAP NiFi Registry
NiFi Registry NiFi. $NIFI_REGISTRY_HOME/conf/identity-providers.xml .
identity-providers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<identityProviders>
<provider>
<identifier>ldap-identity-provider</identifier>
<class>org.apache.nifi.registry.security.ldap.LdapIdentityProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">uid=admin,ou=system</property>
<property name="Manager Password">secret</property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://auth.love.you:10389</property>
<property name="User Search Base">ou=people,dc=love,dc=you</property>
<property name="User Search Filter">(uid={0})</property>
<property name="Identity Strategy">USE_USERNAME</property>
<property name="Authentication Expiration">12 hours</property>
</provider>
</identityProviders>
$NIFI_REGISTRY_HOME/conf/authorizers.xml .
authorizers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizers>
<userGroupProvider>
<identifier>ldap-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">uid=admin,ou=system</property>
<property name="Manager Password">secret</property>
<property name="TLS - Keystore"></property>
<property name="TLS - Keystore Password"></property>
<property name="TLS - Keystore Type"></property>
<property name="TLS - Truststore"></property>
<property name="TLS - Truststore Password"></property>
<property name="TLS - Truststore Type"></property>
<property name="TLS - Client Auth"></property>
<property name="TLS - Protocol"></property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://auth.love.you:10389</property>
<property name="Page Size"></property>
<property name="Sync Interval">1 mins</property>
<property name="Group Membership - Enforce Case Sensitivity">false</property>
<property name="User Search Base">ou=people,dc=love,dc=you</property>
<property name="User Object Class">person</property>
<property name="User Search Scope">ONE_LEVEL</property>
<property name="User Search Filter"></property>
<property name="User Identity Attribute">cn</property>
<property name="User Group Name Attribute">member</property>
<property name="User Group Name Attribute - Referenced Group Attribute"></property>
<property name="Group Search Base">ou=groups,dc=love,dc=you</property>
<property name="Group Object Class">groupOfUniqueNames</property>
<property name="Group Search Scope">ONE_LEVEL</property>
<property name="Group Search Filter"></property>
<property name="Group Name Attribute">cn</property>
<property name="Group Member Attribute"></property>
<property name="Group Member Attribute - Referenced User Attribute"></property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
<property name="User Group Provider">ldap-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">nifiAdmin</property>
<property name="NiFi Group Name"></property>
</accessPolicyProvider>
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers>
NiFi Registry nifiAdmin , ldif-.
Identity Mapping Properties nifi.properties nifi-registry.properties.
, NiFi NiFi Registry.
, "Proxy user requests". , , . , " " " ", , . , , . , . . .
? , NiFi NiFi Registry , . NiFi Registry NiFi , NiFi Java Keystore , , NiFi Registry. , wildcard , CN=*.love.you.
!
, NiFi . , , CN . . , CN cn. NiFi .
. ApacheDS cn . . . : . NiFi , : nifiUser, nifiAdmin *.love.you. cn=....
, , , , .
, , ....mapping.pattern.dn capture group, ....mapping.value.dn . CN=*.love.you *.love.you, ApacheDS.
NiFi NiFi Registry
NiFi . , , NiFi . , .
Operate:
Policies , :
, *.love.you proxy user requests:
Controller Settings:
REGISTRY CLIENTS:
, + , , :
NiFi Registry, , :
USERS :
:
Bucket Test
NiFi. process group Test. process group Version -> Start version control:
:
SAVE process group :
process group .
NiFi NiFi Cli
, process group flow Registry, () NiFi.
NiFi Toolkit NiFi .
: $NIFI_TOOLKIT_HOME/config/reg.properties $NIFI_TOOLKIT_HOME/config/nifi-prod.properties:
NiFi Registry:
reg.properties
baseUrl=https://nifi-re.love.you:8443
keystore=/love/conf/cert/keystore.jks
keystoreType=jks
keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
keyPasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
truststore=/love/conf/cert/truststore.jks
truststoreType=jks
truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
proxiedEntity=nifiAdmin
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
() NiFi:
nifi-prod.properties
baseUrl=https://nifi.love.you:9443
keystore=/love/conf/cert/keystore.jks
keystoreType=jks
keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
keyPasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
truststore=/love/conf/cert/truststore.jks
truststoreType=jks
truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
proxiedEntity=nifiAdmin
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
:
NiFi Registry ID , flow.
flow ID ID flow, () NiFi.
flow;
flow () NiFi.
$NIFI_TOOLKIT_HOME/bin :
:
./cli.sh registry list-buckets -p ../conf/reg.properties
:
# Name Id Description
- ---- ------------------------------------ -----------
1 Test 84744b22-0e2e-4342-b6ab-6658ff0e54ec (empty)
flow 84744b22-0e2e-4342-b6ab-6658ff0e54ec:
./cli.sh registry list-flows -b 84744b22-0e2e-4342-b6ab-6658ff0e54ec -p ../conf/reg.properties
:
# Name Id Description
- ------------------ ------------------------------------ ------------------
1 PG versioning test bc9a3ff4-2d9c-44b0-bd8e-d86b51d3bac6 PG versioning test
flow:
./cli.sh registry list-flow-versions -f bc9a3ff4-2d9c-44b0-bd8e-d86b51d3bac6 -p ../conf/reg.properties
:
Ver Date Author Message
--- -------------------------- --------- ------------------
1 Thu, Apr 22 2021 09:28 UTC nifiAdmin PG versioning test
flow () NiFi:
./cli.sh nifi pg-import -b 84744b22-0e2e-4342-b6ab-6658ff0e54ec -f bc9a3ff4-2d9c-44b0-bd8e-d86b51d3bac6 -fv 1 -p ../conf/nifi-prod.properties
:
f90df705-0178-1000-04fe-45ba27e76f99
- ID process group () NiFi:
, HowTo . , , (, NiFi Registry), "TRACE" NiFi NiFi Registry . , , .
En lo que respecta a CI / CD, no dibujé toda la cadena. Todos tienen sus propios patrones y escenarios, y la canalización en sí sería fuera de tema. Y el ejemplo anterior NiFi Cli, creo, es un buen punto de partida para implementar su propio esquema de entrega.
Gracias a todos los que lo leyeron y espero que el texto haya sido útil.