Wednesday, October 5, 2022
HomeJavaA Concise Information to Utilizing A number of Azure Storage Accounts from...

A Concise Information to Utilizing A number of Azure Storage Accounts from a Single Spring Boot Utility


Spring initiatives usually are opinionated: 80-90% of use instances are dealt with “by default”, and code is usually rather more concise than could be required in any other case on account of Spring’s desire of conference over configuration. These and different “opinions” can lead to dramatically much less code to put in writing and keep and in consequence, extra centered impression.

Within the overwhelming majority of instances the place Azure Storage is used from an utility, there isn’t a compelling benefit to utilizing greater than a single Azure storage account. However there are edge instances, and being able to make use of a number of Azure Storage accounts from a single app – even when we would solely want that functionality round 10% of the time – may present an extremely helpful extension of our storage superpowers.

This text is the results of a collaboration with Shi Li Chen.

It’s all about assets

The Spring Framework defines the Useful resource interface and gives a number of implementations constructed upon Useful resource to facilitate developer entry to low-level assets. With the intention to deal with a specific type of useful resource, two issues are required:

  • Useful resource implementation
  • ResourcePatternResolver implementation

A Spring utility evaluates assets in query utilizing a number of registered resolvers. When the kind of useful resource is recognized, the suitable Useful resource implementation is used to entry and/or manipulate the underlying useful resource.

If the implementations constructed into Spring Framework don’t fulfill your use case, it’s pretty simple so as to add assist for added kinds of assets by defining your individual implementations of AbstractResource and ResourcePatternResolver interfaces.

This text will introduce the Spring Useful resource, assessment Spring Cloud Azure’s implementation of Spring’s Useful resource (particularly with regard to Azure Storage Account concerns and limitations), and think about how you can increase stated implementation to deal with these edge instances during which it might be helpful to entry a number of Azure Storage Accounts from a single Spring Boot utility.

Getting resourceful

We’ve already talked about that the Spring Framework defines a number of helpful Useful resource implementations. As of this writing, the default sorts are:

  • UrlResource
  • ClassPathResource
  • FileSystemResource
  • PathResource
  • ServletContextResource
  • InputStreamResource
  • ByteArrayResource

As talked about earlier, every useful resource could have a corresponding useful resource resolver.

Enabling your Spring Boot utility to make use of a customized Useful resource requires the next actions:

  • Implement the Useful resource interface by extending AbstractResource
  • Implement the ResourcePatternResolver interface to resolve the customized useful resource kind
  • Register the implementation of ResourcePatternResolver as a bean

NOTE: Your resolver should be added to the default useful resource loader’s resolver set utilizing the org.springframework.core.io.DefaultResourceLoader#addProtocolResolver methodology, however this code is current in AbstractAzureStorageProtocolResolver; extending that class to create your implementation accomplishes this in your behalf except you select to override its setResourceLoader methodology.

ResourceLoader makes an attempt to resolve every Useful resource by evaluating its outlined location/format with all registered protocol sample resolvers till a non-null useful resource is returned. If no match is discovered, the Useful resource can be evaluated in opposition to Spring’s built-in sample resolvers.

Spring assets in Spring Cloud Azure

Spring Cloud Azure gives two Spring useful resource and useful resource sample resolver implementations. On this article, we solely talk about the implementation of the Azure Storage Blob useful resource. You may study the supply code for Spring Cloud Azure Sources at Spring Cloud Azure and associated documentation at Useful resource Dealing with.

NOTE: We use Spring Cloud Azure Starter Storage Blob model 4.2.0 for evaluation and experiments.

Implementation of AbstractResource

The summary implementation AzureStorageResource for Spring Cloud Azure primarily defines the format of the Azure storage useful resource protocol and accommodates the distinctive attributes of the Azure Storage Account service, e.g. the container identify and file identify. It is very important notice that AzureStorageResource is decoupled from the Azure Storage SDK.

The Spring Framework interface WritableResource represents the underlying API we construct upon to learn from and write to the Azure Storage useful resource.

summary class AzureStorageResource extends AbstractResource implements WritableResource {

    non-public boolean isAzureStorageResource(@NonNull String location) {
        ......
    }

    String getContainerName(String location) {
        ......
    }

    String getContentType(String location) {
        ......
    }

    String getFilename(String location) {
        ......
    }

    summary StorageType getStorageType();
}

The StorageBlobResource is Spring Cloud Azure Storage Blob’s implementation of the summary class AbstractResource. We will see StorageBlobResource makes use of the BlobServiceClient from the Azure Storage Blob SDK to implement all summary strategies, counting on the service consumer to work together with the Azure Storage Blob service.

public ultimate class StorageBlobResource extends AzureStorageResource {
    non-public ultimate BlobServiceClient blobServiceClient;
    non-public ultimate BlobContainerClient blobContainerClient;
    non-public ultimate BlockBlobClient blockBlobClient;

    public StorageBlobResource(BlobServiceClient blobServiceClient, String location, Boolean autoCreateFiles,
                               String snapshot, String versionId, String contentType) {
        ......
        this.blobContainerClient = blobServiceClient.getBlobContainerClient(getContainerName(location));
        BlobClient blobClient = blobContainerClient.getBlobClient(getFilename(location));
        this.blockBlobClient = blobClient.getBlockBlobClient();
    }

    @Override
    public OutputStream getOutputStream() throws IOException {
        strive {
            ......
            return this.blockBlobClient.getBlobOutputStream(choices);
        } catch (BlobStorageException e) {
            throw new IOException(MSG_FAIL_OPEN_OUTPUT, e);
        }
    }

    ......

    @Override
    StorageType getStorageType() {
        return StorageType.BLOB;
    }
}

Implementation of ResourcePatternResolver

Spring Cloud Azure gives an summary implementation AbstractAzureStorageProtocolResolver. This class incorporates basic processing of the Azure storage useful resource protocol, exposes particular capabilities of the Azure Storage Account service, and provides the requisite logic to the default useful resource loader. Like AzureStorageResource, the AbstractAzureStorageProtocolResolver can also be not coupled to the Azure Storage SDK.

public summary class AbstractAzureStorageProtocolResolver implements ProtocolResolver, ResourcePatternResolver,
    ResourceLoaderAware, BeanFactoryPostProcessor {

    protected ultimate AntPathMatcher matcher = new AntPathMatcher();

    protected summary StorageType getStorageType();

    protected summary Useful resource getStorageResource(String location, Boolean autoCreate);

    protected ConfigurableListableBeanFactory beanFactory;

    protected summary Stream<StorageContainerItem> listStorageContainers(String containerPrefix);

    protected summary StorageContainerClient getStorageContainerClient(String identify);

    @Override
    public void setResourceLoader(ResourceLoader resourceLoader) {
        if (resourceLoader instanceof DefaultResourceLoader) {
            ((DefaultResourceLoader) resourceLoader).addProtocolResolver(this);
        } else {
            LOGGER.warn("Customized Protocol utilizing azure-{}:// prefix is not going to be enabled.", getStorageType().getType());
        }
    }

    @Override
    public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) throws BeansException {
        this.beanFactory = beanFactory;
    }

    @Override
    public Useful resource resolve(String location, ResourceLoader resourceLoader) {
        if (AzureStorageUtils.isAzureStorageResource(location, getStorageType())) {
            return getResource(location);
        }
        return null;
    }

    @Override
    public Useful resource[] getResources(String sample) throws IOException {
        Useful resource[] assets = null;

        if (AzureStorageUtils.isAzureStorageResource(sample, getStorageType())) {
            if (matcher.isPattern(AzureStorageUtils.stripProtocol(sample, getStorageType()))) {
                String containerPattern = AzureStorageUtils.getContainerName(sample, getStorageType());
                String filePattern = AzureStorageUtils.getFilename(sample, getStorageType());
                assets = resolveResources(containerPattern, filePattern);
            } else {
                return new Useful resource[] { getResource(sample) };
            }
        }
        if (null == assets) {
            throw new IOException("Sources not discovered at " + sample);
        }
        return assets;
    }

    @Override
    public Useful resource getResource(String location) {
        Useful resource useful resource = null;

        if (AzureStorageUtils.isAzureStorageResource(location, getStorageType())) {
            useful resource = getStorageResource(location, true);
        }

        if (null == useful resource) {
            throw new IllegalArgumentException("Useful resource not discovered at " + location);
        }
        return useful resource;
    }

    /**
     * Storage container merchandise.
     */
    protected static class StorageContainerItem {
        non-public ultimate String identify;
        ......
    }

    protected static class StorageItem {

        non-public ultimate String container;
        non-public ultimate String identify;
        non-public ultimate StorageType storageType;
        ......
    }

    protected interface StorageContainerClient {

        ......
    }
}

The useful resource resolver AzureStorageBlobProtocolResolver is Spring Cloud Azure Storage Blob’s implementation of ResourcePatternResolver. It encapsulates assets in accordance with the situation or storage merchandise sample primarily based on BlobServiceClient and returns the related StorageBlobResource.

public ultimate class AzureStorageBlobProtocolResolver extends AbstractAzureStorageProtocolResolver {

    non-public BlobServiceClient blobServiceClient;
    @Override
    protected StorageType getStorageType() {
        return StorageType.BLOB;
    }

    @Override
    protected Useful resource getStorageResource(String location, Boolean autoCreate) {
        return new StorageBlobResource(getBlobServiceClient(), location, autoCreate);
    }

    non-public BlobServiceClient getBlobServiceClient() {
        if (blobServiceClient == null) {
            blobServiceClient = beanFactory.getBean(BlobServiceClient.class);
        }
        return blobServiceClient;
    }
}

Opinions

As talked about at first of this publish, the default capabilities fulfill the necessities admirably within the overwhelming majority of circumstances. However in accordance with the Spring ethos, Spring Cloud Azure Starter Storage Blob was designed to seamlessly deal with 80-90% of use instances “out of the field”, whereas nonetheless permitting for remaining (edge) instances with some additional effort.

As written, the storage blob useful resource helps a number of container operations utilizing the identical storage account. The salient level is that the blob paths below totally different containers may be correctly resolved into StorageBlobResource objects. Combining the sooner code for StorageBlobResource, the blob useful resource should maintain a blob service consumer, and if blobServiceClient.getBlobContainerClient(getContainerName(location)) efficiently returns a BlobServiceClient, the blob useful resource may be resolved and retrieved.

The BlobServiceClient bean represents an Azure Storage Account within the Azure Storage Blob SDK, that means that the present implementation doesn’t assist simultaneous availability utilizing a number of Azure Storage Accounts.

Creating an prolonged model of Spring Cloud Azure Starter Storage Blob

For these uncommon instances during which it is likely to be helpful to concurrently entry a number of Azure Storage accounts from the identical utility, there’s a technique to make that occur. To reveal this functionality, let’s create a brand new library referred to as spring-cloud-azure-starter-storage-blob-extend. The one exterior dependency for this new library is the prevailing spring-cloud-azure-starter-storage-blob.

Prolong the Storage Blob properties

Whereas the first purpose is to assist a number of storage accounts, a secondary design purpose is to make use of an identical construction to AzureStorageBlobProperties with a view to decrease the training curve and to retain Spring Cloud Azure 4.0’s out of the field authentication options.

public class ExtendAzureStorageBlobsProperties {

    public static ultimate String PREFIX = "spring.cloud.azure.storage.blobs";

    non-public boolean enabled = true;

    non-public ultimate Listing<AzureStorageBlobProperties> configurations = new ArrayList<>();

    public boolean isEnabled() {
        return enabled;
    }

    public void setEnabled(boolean enabled) {
        this.enabled = enabled;
    }

    public Listing<AzureStorageBlobProperties> getConfigurations() {
        return configurations;
    }
}

Dynamically register Storage Blob beans

Since there can be a number of Storage Account configurations, we should identify the beans corresponding to every storage account. The cleanest strategy is to easily use the account identify because the bean identify.

Now, let’s dynamically register these beans with the Spring context.

@Configuration(proxyBeanMethods = false)
@ConditionalOnProperty(worth = { "spring.cloud.azure.storage.blobs.enabled"}, havingValue = "true")
public class ExtendStorageBlobsAutoConfiguration implements BeanDefinitionRegistryPostProcessor, EnvironmentAware {

    non-public Surroundings atmosphere;

    public static ultimate String EXTEND_STORAGE_BLOB_PROPERTIES_BEAN_NAME = "extendAzureStorageBlobsProperties";

    @Override
    public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) throws BeansException {
        AzureGlobalProperties azureGlobalProperties =
            Binder.get(atmosphere)
                  .bind(AzureGlobalProperties.PREFIX, AzureGlobalProperties.class)
                  .orElse(new AzureGlobalProperties());
        ExtendAzureStorageBlobsProperties blobsProperties =
            Binder.get(atmosphere)
                  .bind(ExtendAzureStorageBlobsProperties.PREFIX, ExtendAzureStorageBlobsProperties.class)
                  .orElseThrow(() -> new IllegalArgumentException("Can't bind the azure storage blobs properties."));
        // merge properties
        for (AzureStorageBlobProperties azureStorageBlobProperties : blobsProperties.getConfigurations()) {
            AzureStorageBlobProperties transProperties = new AzureStorageBlobProperties();
            AzureGlobalPropertiesUtils.loadProperties(azureGlobalProperties, transProperties);
            copyAzureCommonPropertiesIgnoreTargetNull(transProperties, azureStorageBlobProperties);
        }

        DefaultListableBeanFactory manufacturing facility = (DefaultListableBeanFactory) beanFactory;
        registryBeanExtendAzureStorageBlobsProperties(manufacturing facility, blobsProperties);
        blobsProperties.getConfigurations().forEach(blobProperties -> registryBlobBeans(manufacturing facility, blobProperties));
    }

    non-public void registryBeanExtendAzureStorageBlobsProperties(DefaultListableBeanFactory beanFactory,
                                                               ExtendAzureStorageBlobsProperties blobsProperties) {
        BeanDefinitionBuilder beanDefinitionBuilder = BeanDefinitionBuilder.genericBeanDefinition(ExtendAzureStorageBlobsProperties.class,
            () -> blobsProperties);
        AbstractBeanDefinition rawBeanDefinition = beanDefinitionBuilder.getRawBeanDefinition();
        beanFactory.registerBeanDefinition(EXTEND_STORAGE_BLOB_PROPERTIES_BEAN_NAME, rawBeanDefinition);
    }

    non-public void registryBlobBeans(DefaultListableBeanFactory beanFactory, AzureStorageBlobProperties blobProperties) {
        String accountName = getStorageAccountName(blobProperties);
        Assert.hasText(accountName, "accountName can't be null or empty.");
        registryBeanStaticConnectionStringProvider(beanFactory, blobProperties, accountName);
        registryBeanBlobServiceClientBuilderFactory(beanFactory, blobProperties, accountName);
        registryBeanBlobServiceClientBuilder(beanFactory, accountName);
        registryBeanBlobServiceClient(beanFactory, accountName);
        registryBeanBlobContainerClient(beanFactory, blobProperties, accountName);
        registryBeanBlobClient(beanFactory, blobProperties, accountName);
    }

    non-public void registryBeanBlobServiceClientBuilder(DefaultListableBeanFactory beanFactory,
                                                      String accountName) {
        BeanDefinitionBuilder beanDefinitionBuilder = BeanDefinitionBuilder.genericBeanDefinition(BlobServiceClientBuilder.class,
            () -> {
                BlobServiceClientBuilderFactory builderFactory =
                    beanFactory.getBean(accountName + BlobServiceClientBuilderFactory.class.getSimpleName(),
                        BlobServiceClientBuilderFactory.class);
                return builderFactory.construct();
            });
        AbstractBeanDefinition rawBeanDefinition = beanDefinitionBuilder.getRawBeanDefinition();
        beanFactory.registerBeanDefinition(
            accountName + BlobServiceClientBuilder.class.getSimpleName(), rawBeanDefinition);
    }

    ......

    @Override
    public void setEnvironment(Surroundings atmosphere) {
        this.atmosphere = atmosphere;
    }
}

Prolong the AzureStorageBlobProtocolResolver

The subsequent activity is to make any container resolvable by the identical useful resource sample resolver. Specifying a storage blob useful resource location similar to azure-blob-accountname://containername/take a look at.txt, the resolver will use that to find the suitable BlobServiceClient bean by Azure Storage Account identify and return the storage useful resource.

public class ExtendAzureStorageBlobProtocolResolver extends ExtendAbstractAzureStorageProtocolResolver {
    non-public ultimate Map<String, BlobServiceClient> blobServiceClientMap = new HashMap<>();

    @Override
    protected Useful resource getStorageResource(String location, Boolean autoCreate) {
        return new ExtendStorageBlobResource(getBlobServiceClient(location), location, autoCreate);
    }

    non-public BlobServiceClient getBlobServiceClient(String locationPrefix) {
        String storageAccount = ExtendAzureStorageUtils.getStorageAccountName(locationPrefix, getStorageType());
        Assert.notNull(storageAccount, "storageAccount can't be null.");
        String accountKey = storageAccount.toLowerCase(Locale.ROOT);
        if (blobServiceClientMap.containsKey(accountKey)) {
            return blobServiceClientMap.get(accountKey);
        }

        BlobServiceClient blobServiceClient = beanFactory.getBean(
            accountKey + BlobServiceClient.class.getSimpleName(), BlobServiceClient.class);
        Assert.notNull(blobServiceClient, "blobServiceClient can't be null.");
        blobServiceClientMap.put(accountKey, blobServiceClient);
        return blobServiceClient;
    }
}

Once more, it’s essential to add the bean ExtendAzureStorageBlobProtocolResolver to the Spring context.

Testing the Spring Cloud Azure Starter Storage Blob Prolong

You should use begin.spring.io to generate a Spring Boot 2.6.7 or higher challenge with Azure Storage assist (or construct on this storage blob pattern in case you choose).

Add the extending starter dependency to the pom.xml file:

<dependency>
  <groupId>com.azure.spring.prolong</groupId>
  <artifactId>spring-cloud-azure-starter-storage-blob-extend</artifactId>
  <model>1.0-SNAPSHOT</model>
</dependency>


Delete the src/most important/assets/utility.properties file or add the next configuration file application-extend.yml, which allows a number of storage account utilization:

application-extend.yml

spring:
  cloud:
    azure:
      storage:
        blob:
          enabled: false
        blobs:
          enabled: true
          configurations:
            - account-name: ${FIRST_ACCOUNT}
              container-name: ${FIRST_CONTAINER}
              account-key: ${ACCOUNT_KEY_OF_FIRST_ACCOUNT}
            - account-name: ${SECOND_ACCOUNT}
              container-name: ${SECOND_CONTAINER}
              account-key: ${ACCOUNT_KEY_OF_SECOND_ACCOUNT}

NOTE: You could present values for the atmosphere variables above (listed in all capital letters) with energetic Azure Storage Account useful resource data.

Add class com.azure.spring.prolong.pattern.storage.useful resource.prolong.SampleDataInitializer with the next physique:

@Profile("prolong")
@Element
public class SampleDataInitializer implements CommandLineRunner {
    ultimate static Logger logger = LoggerFactory.getLogger(SampleDataInitializer.class);

    non-public ultimate ConfigurableEnvironment env;

    non-public ultimate ExtendAzureStorageBlobProtocolResolver resolver;
    non-public ultimate ExtendAzureStorageBlobsProperties properties;

    public SampleDataInitializer(ConfigurableEnvironment env, ExtendAzureStorageBlobProtocolResolver resolver,
                                 ExtendAzureStorageBlobsProperties properties) {
        this.env = env;
        this.resolver = resolver;
        this.properties = properties;
    }

    /**
     * That is used to initialize some information for every Azure Storage Account Blob container.
     */
    @Override
    public void run(String... args) {
        properties.getConfigurations().forEach(this::writeDataByStorageAccount);
    }

    non-public void writeDataByStorageAccount(AzureStorageBlobProperties blobProperties) {
        String containerName = blobProperties.getContainerName();
        if (!StringUtils.hasText(containerName) || blobProperties.getAccountName() == null) {
            return;
        }

        String accountName = getStorageAccountName(blobProperties);
        logger.data("Start to initialize the {} container of the {} account", containerName, accountName);
        lengthy currentTimeMillis = System.currentTimeMillis();
        String fileName = "fileName-" + currentTimeMillis;
        String information = "information" + currentTimeMillis;
        Useful resource storageBlobResource = resolver.getResource("azure-blob-" + accountName + "://" + containerName +"/" + fileName + ".txt");
        strive (OutputStream os = ((WritableResource) storageBlobResource).getOutputStream()) {
            os.write(information.getBytes());
            logger.data("Write information to container={}, fileName={}.txt", containerName, fileName);
        } catch (IOException e) {
            logger.error("Write information exception", e);
        }
        logger.data("Finish to initialize the {} container of the {} account", containerName, accountName);
    }
}

Run the pattern with following Maven command:

mvn clear spring-boot:run -Dspring-boot.run.profiles=prolong

Lastly, confirm the anticipated final result. Your console ought to show the next output:

c.a.s.e.s.s.r.e.SampleDataInitializer    : Start to initialize the container first of the account firstaccount.
c.a.s.e.s.s.r.e.SampleDataInitializer    : Write information to container=first, fileName=fileName-1656641340271.txt
c.a.s.e.s.s.r.e.SampleDataInitializer    : Finish to initialize the container first of the account firstaccount.
c.a.s.e.s.s.r.e.SampleDataInitializer    : Start to initialize the container second of the account secondaccount.
c.a.s.e.s.s.r.e.SampleDataInitializer    : Write information to container=second, fileName=fileName-1656641343572.txt
c.a.s.e.s.s.r.e.SampleDataInitializer    : Finish to initialize the container second of the account secondaccount.

All pattern challenge code is revealed on the repository spring-cloud-azure-starter-storage-blob-extend-sample.

Inside this prolonged utility, it’s nonetheless potential to revert to the unique, single storage account utilization of Spring Cloud Azure Starter Storage Blob by including the next configuration file application-current.yml:

spring:
  cloud:
    azure:
      storage:
        blob:
          account-name: ${FIRST_ACCOUNT}
          container-name: ${FIRST_CONTAINER}
          account-key: ${ACCOUNT_KEY_OF_FIRST_ACCOUNT}
present:
  second-container: ${SECOND_CONTAINER}

NOTE: You could set or change the listed atmosphere variable assigned values with energetic Azure Storage Account useful resource data.

Run the pattern with following Maven command:

mvn clear spring-boot:run -Dspring-boot.run.profiles=present

To confirm right operation utilizing a single storage account, examine terminal output with that listed right here:

c.a.s.e.s.s.r.c.SampleDataInitializer    : StorageApplication information initialization of 'first-container' start ...
c.a.s.e.s.s.r.c.SampleDataInitializer    : Write information to container=first-container, fileName=fileName1656641162614.txt
c.a.s.e.s.s.r.c.SampleDataInitializer    : StorageApplication information initialization of 'first-container' finish ...
c.a.s.e.s.s.r.c.SampleDataInitializer    : StorageApplication information initialization of 'second-container' start ...
c.a.s.e.s.s.r.c.SampleDataInitializer    : Write information to container=second-container, fileName=fileName1656641165411.txt
c.a.s.e.s.s.r.c.SampleDataInitializer    : StorageApplication information initialization of 'second-container' finish ...

Conclusion

Implementing a selected useful resource kind and corresponding sample resolver is comparatively easy, largely due to clear documentation, the various built-in implementations, frequent utilization inside the Spring know-how stack.

One level that warrants consideration is the protocol definition for the useful resource, e.g. the Azure Storage Blob Useful resource. We should notice whether or not we’re utilizing azure-blob:// or azure-blob-[account-name]:// and plan app capabilities accordingly. Moreover, because the identifier of a community useful resource should be uniquely identifiable, the latter location format might end in a for much longer identify and in addition exposes the identify of the storage account. These tradeoffs should be evaluated in gentle of necessities and threat profile.

References & Helpful Sources

RELATED ARTICLES

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Most Popular

Recent Comments