Clever Geek Handbook

Cómo restaurar Sentry después de una actualización fallida

Hola. Quiero hablar sobre cómo fue la restauración de Sentry después de una actualización fallida.



¿Qué es Sentry ?



Es un sistema de seguimiento de errores completo y de código abierto que admite una amplia gama de lenguajes y entornos móviles nativos, del lado del servidor, del navegador, de escritorio, incluidos PHP, Node. js, Python, Ruby, C #, Java, Go, React, Angular, Vue, JavaScript y más.



Un poco sobre lo simple y sin restricciones que todo se rompió.



Vivimos en la versión 9.0.0 y es hora de actualizar. Habiendo sentido la interfaz web 10.0.0, decidí actualizarla. ¡Este fue el mayor error!



La actualización se llevó a cabo como de costumbre. Primero, vaya a 9.1.2 y luego a 10.0.0 (de lo contrario, es imposible). Entonces descubrí que el trabajador no se inicia. La razón de esto es la adición de nuevas aplicaciones y dependencias.



Los intentos de crear aplicaciones y meterlas en k8s no han tenido éxito.



Se decidió retroceder. Pero aquí quedé decepcionado. Después de migrar a la versión 10, parte de los datos se mueven a clickhouse y se eliminan de Postgres.



Después de 3 horas de trabajo y la participación de un experto en administración de bases de datos, los datos volvieron a funcionar.



¿Cómo?



Creamos una instancia de PG a su lado, eliminamos el esquema de la versión 9.0.0 allí y comenzamos una recuperación tabular. Por supuesto, hubo algunos errores y un archivo.



La historia no termina ahí. Es hora de actualizar a la versión 9.1.2.



Recogió la imagen, ejecute: 



sentry upgrade


y vemos este maravilloso mensaje: 



 ! I'm not trusting myself; either fix this yourself by fiddling
 ! with the south_migrationhistory table, or pass --delete-ghost-migrations
 ! to South to have it delete ALL of these records (this may not be good).
Exception in thread Thread-1 (most likely raised during interpreter shutdown)


Después de una larga búsqueda, prueba y error, se encontró una solución.



Necesita restaurar el esquema de la versión actual.



Para hacer esto, clonamos el repositorio: 



git clone https://github.com/getsentry/onpremise.git


Pasamos a la rama 9.1.2, ya que la rama 9.0.0 ya no está. 



cd onpremise && git checkout tags/9.1.2


A continuación, editamos la imagen en el Dockerfile: 



#cat Dockerfile 
ARG SENTRY_IMAGE
FROM ${SENTRY_IMAGE:-sentry:9.0.0}-onbuild
Luego edite el archivo de redacción



docker-compose.yml 
# NOTE: This docker-compose.yml is meant to be just an example of how
# you could accomplish this on your own. It is not intended to work in
# all use-cases and must be adapted to fit your needs. This is merely
# a guideline.

# See docs.getsentry.com/on-premise/server/ for full
# instructions

version: '3.4'

x-defaults: &defaults
  restart: unless-stopped
  build:
    context: .
  depends_on:
    - redis
   # - postgres
    - memcached
    - smtp
  env_file: .env
  environment:
    SENTRY_MEMCACHED_HOST: memcached
    SENTRY_REDIS_HOST: redis
    SENTRY_POSTGRES_HOST: 'sentry.cl.ats'
    SENTRY_DB_USER: 'sentryDB'
    SENTRY_DB_NAME: 'sentry_user'
    SENTRY_DB_PASSWORD: 'sentry_passwd'
    SENTRY_POSTGRES_PORT: 5432
    SENTRY_EMAIL_HOST: smtp
  volumes:
    - sentry-data:/var/lib/sentry/files


services:
  smtp:
    restart: unless-stopped
    image: tianon/exim4

  memcached:
    restart: unless-stopped
    image: memcached:1.5-alpine

  redis:
    restart: unless-stopped
    image: redis:3.2-alpine

  #postgres:
  #  restart: unless-stopped
  #  image: postgres:9.5
  #  environment:
  #    POSTGRES_HOST_AUTH_METHOD: 'trust'
  #  volumes:
  #    - sentry-postgres:/var/lib/postgresql/data

  web:
    <<: *defaults
    ports:
      - '9000:9000'

  cron:
    <<: *defaults
    command: run cron

  worker:
    <<: *defaults
    command: run worker


volumes:
    sentry-data:
      external: true
    #sentry-postgres:
    #  external: true




Pocas explicaciones. Como tengo una instancia de base de datos separada, omito la creación a través de redactar, y tampoco es necesario el volumen sentry-postgres.



Tras estas manipulaciones, iniciamos el montaje: 



./install.sh


En el proceso, puede aparecer un error, que mencioné anteriormente. No prestamos atención y una vez finalizado el proceso, ejecutamos el siguiente comando: 



docker-compose run --rm web sentry django migrate --delete-ghost-migrations


Conclusión aproximada 
Creating network "onpremise_default" with the default driver
Creating onpremise_memcached_1 ... done
Creating onpremise_smtp_1      ... done
Creating onpremise_redis_1     ... done
21:30:26 [INFO] sentry.plugins.github: apps-not-configured
Running migrations for sentry:
- Nothing to migrate.
 - Loading initial data for sentry.
Installed 0 object(s) from 0 fixture(s)
Running migrations for nodestore:
- Nothing to migrate.
 - Loading initial data for nodestore.
Installed 0 object(s) from 0 fixture(s)
Running migrations for search:
- Nothing to migrate.
 - Loading initial data for search.
Installed 0 object(s) from 0 fixture(s)
Running migrations for social_auth:
- Nothing to migrate.
 - Loading initial data for social_auth.
Installed 0 object(s) from 0 fixture(s)
Running migrations for tagstore:
- Nothing to migrate.
 - Loading initial data for tagstore.
Installed 0 object(s) from 0 fixture(s)
Running migrations for jira_ac:
- Nothing to migrate.
 - Loading initial data for jira_ac.
Installed 0 object(s) from 0 fixture(s)
Running migrations for hipchat_ac:
- Nothing to migrate.
 - Loading initial data for hipchat_ac.
Installed 0 object(s) from 0 fixture(s)




Entonces puede ejecutar la actualización. 



docker-compose run --rm web sentry upgrade


Stdout 
Starting onpremise_smtp_1      ... done
Starting onpremise_memcached_1 ... done
Starting onpremise_redis_1     ... done
21:30:48 [INFO] sentry.plugins.github: apps-not-configured
Syncing...
Creating tables ...
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)

Synced:
 > django.contrib.admin
 > django.contrib.auth
 > django.contrib.contenttypes
 > django.contrib.messages
 > django.contrib.sessions
 > django.contrib.sites
 > django.contrib.staticfiles
 > crispy_forms
 > debug_toolbar
 > raven.contrib.django.raven_compat
 > rest_framework
 > sentry.plugins.sentry_interface_types
 > sentry.plugins.sentry_mail
 > sentry.plugins.sentry_urls
 > sentry.plugins.sentry_useragents
 > sentry.plugins.sentry_webhooks
 > sudo
 > south
 > sentry_plugins.slack

Not synced (use migrations):
 - sentry
 - sentry.nodestore
 - sentry.search
 - social_auth
 - sentry.tagstore
 - sentry_plugins.jira_ac
 - sentry_plugins.hipchat_ac
(use ./manage.py migrate to migrate these)
Running migrations for sentry:
- Nothing to migrate.
 - Loading initial data for sentry.
Installed 0 object(s) from 0 fixture(s)
Running migrations for nodestore:
- Nothing to migrate.
 - Loading initial data for nodestore.
Installed 0 object(s) from 0 fixture(s)
Running migrations for search:
- Nothing to migrate.
 - Loading initial data for search.
Installed 0 object(s) from 0 fixture(s)
Running migrations for social_auth:
- Nothing to migrate.
 - Loading initial data for social_auth.
Installed 0 object(s) from 0 fixture(s)
Running migrations for tagstore:
- Nothing to migrate.
 - Loading initial data for tagstore.
Installed 0 object(s) from 0 fixture(s)
Running migrations for jira_ac:
- Nothing to migrate.
 - Loading initial data for jira_ac.
Installed 0 object(s) from 0 fixture(s)
Running migrations for hipchat_ac:
- Nothing to migrate.
 - Loading initial data for hipchat_ac.
Installed 0 object(s) from 0 fixture(s)
Creating missing DSNs
Correcting Group.num_comments counter




Entonces, el circuito se restaura. Puede actualizar a 9.1.2.



Para hacer esto, en el Dockerfile, cambie la versión a 9.1.2: 



ARG SENTRY_IMAGE
FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild
Y ejecuta install.sh 



./install.sh


Si todo se hace correctamente, el proceso se completará sin errores. La salida debería ser algo como esto:



Stdout 
# ./install.sh 
Checking minimum requirements...

Creating volumes for persistent storage...
Created sentry-data.

.env already exists, skipped creation.

Building and tagging Docker images...

smtp uses an image, skipping
memcached uses an image, skipping
redis uses an image, skipping
Building web
Step 1/2 : ARG SENTRY_IMAGE
Step 2/2 : FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild
# Executing 4 build triggers
 ---> Running in 6c97f9fcaf63
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Removing intermediate container 6c97f9fcaf63
 ---> Running in 9e0f65ee3af6
Removing intermediate container 9e0f65ee3af6
 ---> Running in 09754c44319c
Removing intermediate container 09754c44319c
 ---> a12fa31c2675

Successfully built a12fa31c2675
Successfully tagged onpremise_web:latest
Building cron
Step 1/2 : ARG SENTRY_IMAGE
Step 2/2 : FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild
# Executing 4 build triggers
 ---> Using cache
 ---> Using cache
 ---> Using cache
 ---> Using cache
 ---> a12fa31c2675

Successfully built a12fa31c2675
Successfully tagged onpremise_cron:latest
Building worker
Step 1/2 : ARG SENTRY_IMAGE
Step 2/2 : FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild
# Executing 4 build triggers
 ---> Using cache
 ---> Using cache
 ---> Using cache
 ---> Using cache
 ---> a12fa31c2675

Successfully built a12fa31c2675
Successfully tagged onpremise_worker:latest

Docker images built.

Generating secret key...
Secret key written to .env

Setting up database...
Starting onpremise_smtp_1      ... done
Starting onpremise_redis_1     ... done
Starting onpremise_memcached_1 ... done
21:35:07 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
21:35:10 [INFO] sentry.plugins.github: apps-not-configured
Syncing...
Creating tables ...
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)
Migrating...
Running migrations for sentry:
 - Migrating forwards to 0472_auto__add_field_sentryapp_author.
 > sentry:0424_auto__add_field_integration_status
 > sentry:0425_auto__add_index_pullrequest_organization_id_merge_commit_sha
 > sentry:0425_remove_invalid_github_idps
 > sentry:0426_auto__add_sentryappinstallation__add_sentryapp__add_field_user_is_sent
 > sentry:0427_auto__add_eventattachment__add_unique_eventattachment_project_id_event
 > sentry:0428_auto__add_index_eventattachment_project_id_date_added
 > sentry:0429_auto__add_integrationexternalproject__add_unique_integrationexternalpr
 > sentry:0430_auto__add_field_organizationintegration_status
 > sentry:0431_auto__add_field_externalissue_metadata
 > sentry:0432_auto__add_field_relay_is_internal
 > sentry:0432_auto__add_index_userreport_date_added__add_index_eventattachment_date_
 > sentry:0433_auto__add_field_relay_is_internal__add_field_userip_country_code__add_
 > sentry:0434_auto__add_discoversavedqueryproject__add_unique_discoversavedqueryproj
 > sentry:0435_auto__add_field_discoversavedquery_created_by
 > sentry:0436_rename_projectdsymfile_to_projectdebugfile
 > sentry:0437_auto__add_field_sentryapp_status
 > sentry:0438_auto__add_index_sentryapp_status__chg_field_sentryapp_proxy_user__chg_
 > sentry:0439_auto__chg_field_sentryapp_owner
 > sentry:0440_auto__del_unique_projectdebugfile_project_debug_id__add_index_projectd
 > sentry:0441_auto__add_field_projectdebugfile_data
 > sentry:0442_auto__add_projectcficachefile__add_unique_projectcficachefile_project_
 > sentry:0443_auto__add_field_organizationmember_token_expires_at
 > sentry:0443_auto__del_dsymapp__del_unique_dsymapp_project_platform_app_id__del_ver
 > sentry:0444_auto__add_sentryappavatar__add_field_sentryapp_redirect_url__add_field
 > sentry:0445_auto__add_promptsactivity__add_unique_promptsactivity_user_feature_org
 > sentry:0446_auto__add_index_promptsactivity_project_id
 > sentry:0447_auto__del_field_promptsactivity_organization__add_field_promptsactivit
 > sentry:0448_auto__add_field_sentryapp_is_alertable
 > sentry:0449_auto__chg_field_release_owner
 > sentry:0450_auto__del_grouphashtombstone__del_unique_grouphashtombstone_project_ha
 > sentry:0451_auto__del_field_projectbookmark_project_id__add_field_projectbookmark_
 > sentry:0452_auto__add_field_sentryapp_events
 > sentry:0452_auto__del_field_releaseenvironment_organization_id__del_field_releasee
 > sentry:0453_auto__add_index_releasefile_release_name
 > sentry:0454_resolve_duplicate_0452
 > sentry:0455_auto__add_field_groupenvironment_first_seen
 > sentry:0456_auto__add_dashboard__add_unique_dashboard_organization_title__add_widg
 > sentry:0457_auto__add_field_savedsearch_is_global__chg_field_savedsearch_project__
 > sentry:0457_auto__add_monitorcheckin__add_monitor__add_index_monitor_type_next_che
 > sentry:0458_global_searches_data_migration
Saved Searchs: 100% |########################################################################################################################################################################| Time: 0:00:07
 > sentry:0459_global_searches_unique_constraint
 > sentry:0460_auto__add_field_servicehook_organization_id
 > sentry:0461_event_attachment_indexes
 > sentry:0462_auto__add_servicehookproject
 > sentry:0462_releaseenvironment_project_id
 > sentry:0463_backfill_service_hook_project
 > sentry:0464_auto__add_sentryappcomponent__add_field_sentryapp_schema
 > sentry:0464_groupenvironment_foreignkeys
 > sentry:0465_sync
 > sentry:0466_auto__add_platformexternalissue__add_unique_platformexternalissue_grou
 > sentry:0467_backfill_integration_status
 > sentry:0468_auto__add_field_projectdebugfile_code_id__add_index_projectdebugfile_p
 > sentry:0468_recent_search
 > sentry:0469_fix_state
 > sentry:0470_org_saved_search
 > sentry:0471_global_saved_search_types
 > sentry:0472_auto__add_field_sentryapp_author
The following content types are stale and need to be deleted:

    sentry | dsymobject
    sentry | dsymapp
    sentry | useridentity
    sentry | dsymsymbol
    sentry | dsymsdk
    sentry | globaldsymfile
    sentry | versiondsymfile
    sentry | projectdsymfile
    sentry | grouphashtombstone
    sentry | minidumpfile
    sentry | dsymbundle

Any objects related to these content types by a foreign key will also
be deleted. Are you sure you want to delete these content types?
If you're unsure, answer 'no'.

    Type 'yes' to continue, or 'no' to cancel: yes
Running migrations for sentry.nodestore:
- Nothing to migrate.
Running migrations for sentry.search:
- Nothing to migrate.
Running migrations for social_auth:
- Nothing to migrate.
Running migrations for sentry.tagstore:
- Nothing to migrate.
Running migrations for sentry_plugins.hipchat_ac:
- Nothing to migrate.
Running migrations for sentry_plugins.jira_ac:
- Nothing to migrate.

Synced:
 > django.contrib.admin
 > django.contrib.auth
 > django.contrib.contenttypes
 > django.contrib.messages
 > django.contrib.sessions
 > django.contrib.sites
 > django.contrib.staticfiles
 > crispy_forms
 > debug_toolbar
 > rest_framework
 > sentry.plugins.sentry_interface_types
 > sentry.plugins.sentry_mail
 > sentry.plugins.sentry_urls
 > sentry.plugins.sentry_useragents
 > sentry.plugins.sentry_webhooks
 > sudo
 > south

Migrated:
 - sentry
 - sentry.nodestore
 - sentry.search
 - social_auth
 - sentry.tagstore
 - sentry_plugins.hipchat_ac
 - sentry_plugins.jira_ac
Creating missing DSNs
Correcting Group.num_comments counter
Cleaning up...

----------------
You're all done! Run the following command get Sentry running:

  docker-compose up -d




Eso es todo, la versión 9.1.2 está instalada.



Conclusiones.



Estas son verdades comunes conocidas por todos. Tómese su tiempo, haga copias de seguridad y verifique todo varias veces.



¡Gracias por su atención!


More articles:


All Articles