Elite: Dangerous y CosmosDB

o7 cmdr!

En una cálida noche de cuarentena, en uno de los chats de telegramas sobre Elite: Dangerous, estalló una discusión sobre el tema: ¿qué tipo de estrellas tienen planetas similares a la Tierra con mayor frecuencia?

El caso es que la exploración planetaria es una de las principales mecánicas del juego. Y en la jerarquía de la utilidad de los planetas, los parecidos a la Tierra están en la cima. Pero su rareza también es bastante alta. Entonces, los comandantes querían saber: ¿a qué estrellas prestar atención cuando se mueva por la Vía Láctea?

De esta discusión nació todo un proyecto, que finalmente enterré. No, encontramos la respuesta a la pregunta planteada. Pero no me gustó el proyecto por varias razones, y después de unos meses de procrastinación, lancé la segunda versión. Lo que resultó de esto, así como la respuesta a la pregunta planteada, en este artículo.

Introducción

En la élite, una gran parte del juego ocurre, lo que sea que piense, en un navegador con una docena de pestañas abiertas. No discutamos bien o no, pero, definitivamente, esto tiene su propio encanto y a muchas personas les gusta.

, , . , Frontier Developments . , .

? , Log- . .

() ( Windows):

C:\Users\User Name\Saved Games\Frontier Developments\Elite Dangerous\

. . .. PC ( ) — .

, , , . ? - , ( , , , , POI, etc).

PC:

. - . , . , , . , 400 (, ! ). , -.

, . ( , ). : , , (, ) .. , , , , — EDSM EDDB.

, , — . .

EDSM , . , , . , , — JSON 10. - - . , , - , . , ! .

% . 6 . .

, , . , , !

, , , - ( , ). EDSM, EDDB, etc. — EDDN. , , , .

, EDDN ? ...

EDDN

EDDN — :

The Elite: Dangerous Data Network is a system for willing Commanders to share dynamic data about the galaxy with others.

By pooling data in a common format, tools and analyses can be produced that add an even greater depth and vibrancy to the in-game universe.



EDDN is not run by or affiliated with Frontier Developments.

( ).

HTTP Endpoint https://eddn.edcd.io:4430/upload/ ( ).

. ZeroMQ tcp://eddn.edcd.io:9500. .

( ):

, . ? , . , ( ) ! , - .

, :

— Microsoft Azure. - . — Azure, dotnet core/standard

--- , .

, :

1. Message Distributor

EDDN Channel . ( schemaRef , ) — . EDDN 5:

• https://eddn.edcd.io/schemas/journal/1
• https://eddn.edcd.io/schemas/blackmarket/1
• https://eddn.edcd.io/schemas/commodity/3
• https://eddn.edcd.io/schemas/outfitting/2
• https://eddn.edcd.io/schemas/shipyard/2

. ? , ( ), , . journal, .. ( ). , , .

, MessageDistributor EDDN . .

2. Azure Storage Queue

( Message Processor , ). , . Storage Account ( ) connection string ( , Azure AD, ). Storage Account ( — , , ).

Azure Storage Emulator Azure Storage Explorer

3. Message Processor

, , . Azure Function App.

, - ? .

, Azure Function WebJob ( , ), . , Azure Function Runtime ( , ) — — , — . , .. — .

, , (Scale Out), , , - , CPU/Memory consumption, etc.

, (in/out bindings). , , QueueTrigger — (, dotnet, ). CosmosDBTrigger . ( ). , , CosmosDB ( ) . - , , : db client ( in-built DI, ). , , queue .

QueueTrigger. , — (invisible) . , 30 ( ). 30 — DequeueCount 1. ( -> visible state on, ++DequeueCount). DequeueCount = 5 ( ), -poison. journal, 5 , journal-poison. ( - ). , . .

: WebJob AppService. — Function App. : , .

. : , , 2 . 2 , , . , (upsert — update || insert) , , ;-) 2 , , . , . , . , , , . , - (, , CosmosDB ACID compliant).

IMHO: , . . 1. , — . (, ) .

Cosmos DB Optimistic Concurrency, . . .

4. CosmosDB

. , ( ) , . , .. .

4: Signals, Systems, Stations, Bodies. , journal ( , ? -).

, journal Event, : CarrierJump, Docked, FsdJump, Location, SaaSignalsFound, Scan ( ). , — . : — . .

CosmosDB. ( , ) Request Unit per second — RU/s. .. RU/s . 2 : Provisioned throughput Serverless ( , ). , Provisioned RU/s ( ), Serverless, RU/s , , .



Provisioned RU/s. , RU/s 400, 10% 1000% (40-4000 RU/s)



RU/s Provisioned mode — 400. , , , 250 RU/s 150 , Serverless .



, ( CosmosDB). : CosmicClone. Serverless . 2 . Provisioned Mode , , .



, zero downtime "" — , MessageDistributor , . , . . .



CosmosDB

, , : EDDN , .

, React Native — dotnet, . .

: EDDNConsumer. , , .

:

• EDDNConsumerCore — MessageDistributor. EDDN
• EDDNModels — EDDN
• JournalContributor — MessageProcessor journal
• SharedLibrary — . .

EDDNConsumerCore

dotnet core 3.1 . Main DI. HostedService ConsumerService

services.AddHostedService<ConsumerService>();

nuget- ( ): NetMQ — ZeroMq Ionic.Zlib — NetMQ

StartAsync, — NetMQRuntime ClientAsync ( , ):

private async Task ClientAsync()
{
    var utf8 = new UTF8Encoding();
    using (var client = new SubscriberSocket())
    {
        client.Connect(_eddnClientSettings.ConnectionString);
        client.SubscribeToAnyTopic();
        while (true)
        {
            try
            {
                (var bytes, _) = await client.ReceiveFrameBytesAsync();
                var uncompressed = ZlibStream.UncompressBuffer(bytes);
                var result = utf8.GetString(uncompressed);
                await _messageDistributor.DistributeAsync(result);
                _logger.LogInformation(result);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error reading message queue");
            }
        }
    }
}

, ( , 1 ), _messageDistributor , , . .

MessageDistributor DistributeAsync:

public async Task DistributeAsync(string message)
{
    try
    {
        using var stringReader = new StringReader(message);
        using var jsonReader = new JsonTextReader(stringReader);
        var result = _serializer.Deserialize<Entity<BaseMessage>>(jsonReader);
        var queue = await _messageQueueFactory.GetQueueAsync(result);
        await queue.SendMessageAsync(message.Base64Encode());
    }
    catch(Exception ex)
    {
        _logger.LogError(ex, "Error distributing message");
    }
}

, , MessageQueueFactory. — . : . , schemaRef header ( test_data.json ). , , .

, ( — JSON) , . \\ , — , , null default. ( ) , POCO , , , , - . .

, , json- NewtonsoftJson ( json, ). Missing Member

_serializer.MissingMemberHandling = MissingMemberHandling.Error;
_serializer.Error += _serializer_Error;

, Application Insights, Notification Alerts . , , Header. . , .. , missing property. , , .

, header , . , ( ):

public class MessageQueueFactory : IMessageQueueFactory
{
    private readonly StorageAccount _storageOptions;
    private readonly QueueMapping _queueMapping;
    private readonly IDictionary<string, QueueClient> _queues = new Dictionary<string, QueueClient>();

    public MessageQueueFactory(
        IOptions<StorageAccount> storageOptions,
        IOptions<QueueMapping> queueMapping)
    {
        _storageOptions = storageOptions.Value;
        _queueMapping = queueMapping.Value;
    }

    public async Task<QueueClient> GetQueueAsync(Entity<BaseMessage> entity)
    {
        if (_queueMapping.TryGetValue(entity.SchemaRef, out var queueName))
        {
            if (!_queues.ContainsKey(queueName))
            {
                var client = new QueueClient(_storageOptions.StorageConnectionString, queueName);
                await client.CreateIfNotExistsAsync();
                _queues.Add(queueName, client);
            }
            return _queues[queueName];
        }
        throw new ArgumentException($"Queue {entity.SchemaRef} has not configured", nameof(entity.SchemaRef));
    }
}

, appsettings json :

{
  "QueueMapping": {
    "eddn.edcd.io/schemas/journal/1": "journal"
  }
}

( ):

"QueueMapping": {
    "eddn.edcd.io/schemas/journal/1": "journal",
    "eddn.edcd.io/schemas/blackmarket/1": "blackmarket",
    "eddn.edcd.io/schemas/commodity/3": "commodity",
    "eddn.edcd.io/schemas/outfitting/2": "outfitting",
    "eddn.edcd.io/schemas/shipyard/2": "shipyard"
  }

Dictionary<string, string> schemaRef. .

1 _queues — QueueClient, . , . , . round trip , . , , . , .

, — , — . . MessageDistributor, .

. 3 . .

EDDNModels

POCO . , . , JournalMessage:

[JsonProperty("id")]
public override string Id
{
    get => Event switch
    {
        JournalEvent.FsdJump => StarSystem,
        JournalEvent.Scan => BodyName,
        JournalEvent.Docked => StationName,
        JournalEvent.Location => BodyName,
        JournalEvent.CarrierJump => StarSystem,
        JournalEvent.SaaSignalsFound => BodyName,
        _ => $"UnknownID_{Guid.NewGuid()}"
    };
}

Id, , CosmosDB. ORM. , journal ? , . Event Id. - , - , - .

. : , , , , , , , , , \ .. ( JournalMessage). , , .. POCO — .

.

JournalContributor

"" journal — Azure Function App. Function App dotnet core class library , , . FunctionName . : — Function App ( ). - — . , . . , , ( ).

, :

public class JournalContributor
    {
        private readonly IEventTypeProcessorFactory _eventTypeProcessorFactory;

        public JournalContributor(IEventTypeProcessorFactory eventTypeProcessorFactory)
        {
            _eventTypeProcessorFactory = eventTypeProcessorFactory;
        }

        [FunctionName("ContributeJournal")]
        public async Task Run(
            [QueueTrigger("journal", Connection = "AzureWebJobsStorage")]
            Entity<JournalMessage> myQueueItem,
            ILogger log)
        {
            try
            {
                var eventProcessor = _eventTypeProcessorFactory.GetProcessor(myQueueItem.Message);
                await eventProcessor.ProcessEventAsync(myQueueItem.Message);
            }
            catch(Exception ex)
            {
                log.LogError(ex, $"Error processing queue item: {JsonConvert.SerializeObject(myQueueItem)}");
                throw;
            }
        }
    }

QueueTrigger ( ). , Event … .

, , . — , dotnet . Startup Configure, FunctionStartup (assembly):

using JournalContributor.Settings;
using Microsoft.Extensions.Configuration;
using System.IO;

[assembly: FunctionsStartup(typeof(JournalContributor.Startup))]

namespace JournalContributor
{
    public class Startup : FunctionsStartup
    {
        private IConfigurationRoot _functionConfig;
        private readonly string COSMOS_CONNECTION_STRING = Environment.GetEnvironmentVariable("CosmosDBConnectionString");
        private readonly string ENVIRONMENT = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");

        public override void Configure(IFunctionsHostBuilder builder)
        {
            _functionConfig = new ConfigurationBuilder()
                .AddJsonFile(Path.Combine(builder.GetContext().ApplicationRootPath, "appsettings.json"), optional: true, reloadOnChange: true)
                .AddJsonFile(Path.Combine(builder.GetContext().ApplicationRootPath, $"appsettings.{ENVIRONMENT}.json"), optional: true, reloadOnChange: true)
                .Build();

            builder.Services.AddSingleton<CosmosClient>(factory => new CosmosClient(COSMOS_CONNECTION_STRING));
            builder.Services.AddSingleton<IEventTypeProcessorFactory, EventTypeProcessorFactory>();
            builder.Services.AddTransient<FsdJumpProcessor>();
            builder.Services.AddTransient<ScanProcessor>();
            builder.Services.AddTransient<DockedProcessor>();
            builder.Services.AddTransient<LocationProcessor>();
            builder.Services.AddTransient<SaaSignalsFoundProcessor>();
            builder.Services.Configure<CosmosDbSettings>(_functionConfig.GetSection("CosmosDbSettings"));
        }
    }
}

DI . , CosmosClient : (CosmosDB). ? . 2 CosmosDB — , . , .

. , , , switch :

public IEventTypeProcessor GetProcessor(JournalMessage journalMessage) => journalMessage.Event switch
        {
            JournalEvent.FsdJump => _serviceProvider.GetService<FsdJumpProcessor>(),
            JournalEvent.Scan => _serviceProvider.GetService<ScanProcessor>(),
            JournalEvent.Docked => _serviceProvider.GetService<DockedProcessor>(),
            JournalEvent.Location => _serviceProvider.GetService<LocationProcessor>(),
            JournalEvent.SaaSignalsFound => _serviceProvider.GetService<SaaSignalsFoundProcessor>(),
            JournalEvent.CarrierJump => _serviceProvider.GetService<DockedProcessor>(),
            _ => throw new ArgumentException($"Unknown Event {journalMessage.Event}", nameof(journalMessage.Event))
        };

, , . ScanProcessor, .. 2- :

public async Task ProcessEventAsync(JournalMessage message)
        {
            var existingItem = await message.CheckIfItemExists(_bodiesContainer, _options.BodiesCollection.PartitionKey);
            if (existingItem == null)
                await _bodiesContainer.UpsertItemAsync(message);
            else
            {
                //Basic scan type contains less information then others.
                //We`re skipping item upsert if remote item has scan type higher then Basic
                //We`re also skippings if remote item scan type is Detailed (as it`s a maximum info scan)
                //Will update item if both (remote and current) have scan type Basic
                //And will upsert item in case of current item have higher scan type then remote
                if ((message.ScanType == ScanType.Basic && existingItem.ScanType != ScanType.Basic) ||
                    existingItem.ScanType == ScanType.Detailed)
                    return;
                else
                    await _bodiesContainer.UpsertItemAsync(message);
            }
        }

, , ( !) , , , , . . . , . , : , Bodies Id . , .

UpSert, Insert, .. concurrency : Item with such Id already exists ( - ). , , Optimistic Concurrency . . upsert.

, ScanType. Detailed — . Basic — — . Basic — . .

. , 3 , MessageProcessor journal .

, . , Azure WebJob EDDN , Storage Account , Azure Function App Cosmos DB . Application Insights (, -, , ). , .

continuous, .. .

![image](img src="https://habrastorage.org/webt/i_/5u/zq/i_5uzqxkanblmotxacqhtwnp_wq.png)

, , schemaRef . , .

Stream-log : , , … .

. , , .

RU/s . — Provisioned Mode 400 RU/s. , . ~105RU/s.

. 145 . EDSM, .

. Check.

. . .

. . - .

45 . , MSDN , 45 . . , . . 45 — 1.5 . , — .

— . - — , - — "" . .

. ~10 . appinsjournalweprivate . Function App. — , . \. :

( ), host.json JournalContributor:

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingExcludedTypes": "Request",
      "samplingSettings": {
        "isEnabled": true
      }
    },
    "logLevel": {
      "default": "Information",
      "Host": "Error",
      "Function": "Error",
      "Host.Aggregator": "Information"
    }
  }
}

:

, ( 21 ):

( 193 ). - / . 21 1.75 . — appinsweprivate — WebJob. 0.98 0.19 21 .

cdbweprivate — CosmosDB Provisioned Mode 400 RU/s. . cdbslweprivate — Serverless mode CosmosDB. 4.47 .

— laweprivate. Log Analytics Workspace, , , .. CosmosDB. , , , . .

, 175,8 . , .

, . -. appinsights — 3 . , . , % , . - , , 3 — ( ).

-. CosmosDB RU/s ( , ) RU/s (Pricing), :

EDDN. , , Elite: Dangerous. - " " — . , , DLC Odyssey , . .

, Function App Storage Account, , .

, . — .

. .

.

" " " ", . - , . =) , -, ...

— . , - , 145 , EDSM 57 . - - , .

, , , . . " " — .

, — . ? . , , EDSM . , JSON — . — ( ). RU/s , , . .

CosmosDB — Jupyter Notebooks. C#, Python, - . , .

, , . ( , Sol [0,0,0]:

? :

5 :

EDSM, - , . , , — - . , , .

CosmosDB, Azure Function Change Feed ( ), .

— CosmosDB .

Para aquellos que han resistido hasta el final y todavía están interesados ​​en dónde buscar planetas similares a la Tierra en la élite, aquí está su respuesta (los resultados se basan en datos obtenidos de un volcado completo de EDSM hace unos seis meses):

Los tipos F, K, G, A y, por alguna razón, los neutrones constituyen los 5 tipos principales de estrellas en las que los pilotos han encontrado planetas similares a la Tierra. Bueno, estos son los resultados ¯ \ _ (ツ) / ¯

Bien. Eso es todo.

¡Vuela seguro, cmdr!