Continuous Delivery - software-architectural topics

In my last post about Continuous Delivery was the focus on the delivery process. However in this post we will concentrate on the software and the necessary architectural requirements.

So the central question is, how to deliver new functionality as a software many times during a day without adverse effects. How is it possible that companies like Facebook despite major projects with hundreds of developers can manage this?

The first step in the praxis should be the system redesign from a monolithic application towards to split into small pieces (functions, components).
The components are separated and the communication is possible just via a clear API, like for example REST.


The second step would be the redesign of the underlying database. 
Encapsulating the access logic in an interface promotes a service oriented design. A distinction is made between an Application Database, belonging to one application, and Integration Database, shared by different applications and systems.


Avoid Server-side Session State
If short-time information are stored in a local cache of the server then we are talking about Server-side Session State and as such there are the particular instance fails the session is gone. For example during a deployment of new software version to this server instance.
Stateless services enable  a downtime-free deployments.

Keep backward Compatibility
This point is not so easy and it is very complicated to guaranty the compatibility after each modification of the system. Actually this means that the new functionality is co-exists with the old one.

Design for Failure
No system has got 100 percent availability. More important for the enduser is the impact of not functioning component. If one specific function fails, it should not cause the failure of the whole system. 
This is very crucial for continuous delivery concept, otherwise the risk of failure and its impact would be enormous.

Schema-less Database
Changes on the structure of database or its migration are in most of the time very complicated and risky.  The clear advantage is coming with NoSQL databases, or with migration running inside of the application and not in separated database migration scripts. Such scripts last too long and the migration would be not possible in the real-time.


Overall, the goal should be to reach the "Shared-Nothing" Architecture.  This architecture is defined through no direct references between two or many systems which enables a horizontal scalability and release deployment without down-time.

 
 

Read more

Splunk - the swiss army knife of big data

techweekeurope.co.uk

What is it?

Splunk Inc. is a vendor of a solution to monitor, search, analyze, visualize and act on massive streams of real-time and historical machine data.

The solution is widely in use by more than 4,800 enterprises in over 85 countries. The added value of the usage Splunk software is to gain operational intelligence that informs business and customer understanding, improves service and up-time, reduces cost and mitigates cyber-security risk.

Enterprises’ IT systems and infrastructure contain websites, applications, servers, networks, sensors, mobile devices and similar - All of them generate massive amounts of machine data.

The machine data can be turned into valuable insights with help of Splunk.



On the basis of these analyses it is possible to determine problems and investigate security incidents in shorter time than hours or days. End-to-end monitoring of the infrastructure increase the quality of the service and avoids service degradation or outages. Gain real-time visibility into customer experience, transactions and behavior.

Splunk bridges the gap between simple log management and security information and event management (SIEM) products from other vendors.

What separates out Splunk from the world of Syslog servers and SIEM tools is Splunk Apps, a library of add-ons and plugins that make Splunk smarter about particular types of log information, change its look-and-feel, or add new types of domain specific analysis.

If you want to make Splunk work, you've got to be ready to abandon the slick GUI and dive deep into difficult technical configuration, editing configuration files, writing regular expressions, and taking the time to understand where your data are coming from and how Splunk will see them.

How does it work?

Splug goes through of log / data files, parsing and indexing them. After this different searches can be run via the Splunk interface. Splunk supports technologies like Big-data based on Hadoop. With the help of Map Reduce it identifies the events from the log records, and counts the number of occurrence.

de.splunk.com

There are different possible graphical dashboards, that could be built. The result data can be exported via the API to excel for additional regressions and analysis.

Horizontal scaling is fully supported and is the way to go for data processing with Splunk. The bottleneck of the data processing is usually CPU and challenging parameter is RAM. Splunk Enterprise has built in ability to handle a clustered topology.

How much does it cost?

The software licensing model works like this. The trial version included full functionality and can be used for 60 days. Only Splunk Enterprise works as a distributed system.

A Splunk Enterprise License for up to 500mb of indexed data costs $500/mo, and scales up with the organization’s indexing needs. The license is only based on the indexed data, not on the number of queries into the data.

Nice thing is that it does not matter how many instances (machines) are in a cluster the price of the license stays the same!

Read more