Tool for debugging Java Stack Traces – Untangle Threads

FastThreadAnalyzer is the tool which helps to debug you java stack traces.

It provides various insights about the stacks , club the similar stack together and provide you direction or in some cases the exact issue.

Some of the screenshots of the tools:

Threads of same thread pool grouped via thread state:

provides a clear view and you can deduce very easily that whether all threads of a particular pool should be busy or not

Now you see these threads group via similar stacks , which provides great insights on what type of flow are stuck:

Overall Thread State Wise Grouping:

Thread Dependencies:

very easily you can see because of which thread ids these threads are blocked.

There are various other type of grouping of stacks as well like based on length of stacks , deadlocks and so on.

If your org works on java that this should be the default tool to debug the application issues.

Bulk Updation/Insertion of Database Tables in Java using Hibernate – Optimized Way

Hibernate is the most popular orm framework used to interact with databases in java . In this article  we will see what are the various ways using which bulk selection and updation in any table can be done and what is the most effective way when using the hibernate framework in java . 

I  experimented with three ways which are as follows : 

  • Using Hibernate’s Query.list() method.
  • Using ScrollableResults with FORWARD_ONLY scroll mode.
  • Using ScrollableResults with FORWARD_ONLY scroll mode in a StatelessSession.

To decide which one gives best performance for our use case, following tests i performed using the above three ways listed.

  • Select and update 1000 rows.

Let’s see the Code and results by applying above three ways to the operation stated above one by one. 

Using Hibernate’s Query.list() method.

Code Executed : 

   List rows;
        Session session = getSession();
        Transaction transaction = session.beginTransaction();
        try {
            Query query = session.createQuery("FROM PersonEntity WHERE id > :maxId ORDER BY id").setParameter("maxId",
                    MAX_ID_VALUE);
            query.setMaxResults(1000);
            rows = query.list();
            int count = 0;
            for (Object row : rows) {
                PersonEntity personEntity = (PersonEntity) row;
                personEntity.setName(randomAlphaNumeric(30));
                session.saveOrUpdate(personEntity);
                //Always flush and clear the session after updating 50(jdbc_batch_size specified in hibernate.properties) rows
                if (++count % 50 == 0) {
                    session.flush();
                    session.clear();
                }
            }
        } finally {
            if (session != null && session.isOpen()) {
                transaction.commit();
                session.close();
            }
        }

Tests Results : 

  • Time taken:- 360s to 400s
  • Heap Pattern:- gradually increased from 13m to 51m(from jconsole). 

Using ScrollableResults with FORWARD_ONLY scroll mode.

With this we are expecting that it should consume less memory that the 1st approach . Let’s see the results 

Code Executed : 

Session session = getSession();
        Transaction transaction = session.beginTransaction();
        ScrollableResults scrollableResults = session
                .createQuery("FROM PersonEntity WHERE id > " + MAX_ID_VALUE + " ORDER BY id")
                .setMaxResults(1000).scroll(ScrollMode.FORWARD_ONLY);
        int count = 0;
        try {
            while (scrollableResults.next()) {
                PersonEntity personEntity = (PersonEntity) scrollableResults.get(0);
                personEntity.setName(randomAlphaNumeric(30));
                session.saveOrUpdate(personEntity);
                if (++count % 50 == 0) {
                    session.flush();
                    session.clear();
                }
            }
        } finally {
            if (session != null && session.isOpen()) {
                transaction.commit();
                session.close();
            }
        }

Tests Results : 

  • Time taken:- 185s to 200s
  • Heap Pattern:- gradually increased from 13mb to 41mb (measured same using jconsole)

Using ScrollableResults with FORWARD_ONLY scroll mode in a StatelessSession.

A stateless session does not implement a first-level cache nor interact with any second-level cache, nor does it implement transactional write-behind or automatic dirty checking, nor do operations cascade to associated instances. Collections are ignored by a stateless session. Operations performed via a stateless session bypass Hibernate’s event model and interceptors.   

These type of session is always recommended in case of bulk updation as we really do not need these overheads of hibernate features in these type of usecases . 

Code Executed : 

 StatelessSession session = getStatelessSession();
        Transaction transaction = session.beginTransaction();
        ScrollableResults scrollableResults = session
                .createQuery("FROM PersonEntity WHERE id > " + MAX_ID_VALUE + " ORDER BY id")
                .setMaxResults(TRANSACTION_BATCH_SIZE).scroll(ScrollMode.FORWARD_ONLY);
        try {
            while (scrollableResults.next()) {
                PersonEntity personEntity = (PersonEntity) scrollableResults.get(0);
                personEntity.setName(randomAlphaNumeric(20));
                session.update(personEntity);
            }
        } finally {
            if (session != null && session.isOpen()) {
                transaction.commit();
                session.close();
            }
        }

Tests Results : 

  • Time taken:- 185s to 200s
  • Heap Pattern:- gradually increased from 13mb to 39mb

I also performed the same tests with 2000 rows and the results obtained were as follows:-

Results:-

  • Using list():- time taken:- approx 750s, heap pattern:- gradually increased from 13mb to 74 mb
  • Using ScrollableResultSet:- time taken:- approx 380s, heap pattern:- gradually increased from 13mb to 46mb
  • Using Stateless:- time taken:- approx 380s, heap pattern:- gradually increased from 13mb to 43mb

Blocker Problem with all above approaches Tried

ScrollableResults and Stateless ScrollableResults give almost the same performance which is much better than Query.list(). But there is still one problem with all the above approaches. Locking, all the above approaches select and update the data in same transaction, this means for as long as the transaction is running, the rows on which updates have been performed will be locked and any other operations will have to wait for the transaction to finish.

Solution : 

There are two things which we should do here to solve above problem : 

  •  we need to select and update data in different transactions.
  • And updation of these types should be done in Batches

So again I performed the same tests as above but this time update was performed in a different transaction which was commited in batches of 50.

Note:- In case of Scrollable and Stateless we need a different session also, as we need the original session and transaction to scroll through the results.

Results using Batch Processing

  • Using list():- time taken:- approx 400s, heap pattern:- gradually increased from 13mb to 61 mb
  • Using ScrollableResultSet:- time taken:- approx 380s, heap pattern:- gradually increased from 13mb to 51mb
  • Using Stateless:- time taken:- approx 190s, heap pattern:- gradually increased from 13mb to 44mb

Observation:- This temporal performance of ScrollableResults dropped down to become almost equal to Query.list(), but performance of Stateless remained almost same.

Summary and Conclusion

As from all the above experimentation  , in cases where we need to do bulk selection and updation, the best approach in terms of memory consumption and time is as follows : 

  • Use ScrollableResults in a Stateless Session.
  • Perform selection and updation in different transactions in batches of 20 to 50 (Batch Processing) (Note -*-  Batch size  can depend on the case to case basis)

  Sample Code with the best approach

  StatelessSession session = getStatelessSession();
        Transaction transaction = session.beginTransaction();
        ScrollableResults scrollableResults = session
                .createQuery("FROM PersonEntity WHERE id > " + MAX_ID_VALUE + " ORDER BY id")
                .setMaxResults(TRANSACTION_BATCH_SIZE).scroll(ScrollMode.FORWARD_ONLY);
        int count = 0;
        try {
            StatelessSession updateSession = getStatelessSession();
            Transaction updateTransaction = updateSession.beginTransaction();
            while (scrollableResults.next()) {
                PersonEntity personEntity = (PersonEntity) scrollableResults.get(0);
                personEntity.setName(randomAlphaNumeric(5));
                updateSession.update(personEntity);
                if (++count % 50 == 0) {
                    updateTransaction.commit();
                    updateTransaction = updateSession.beginTransaction();
                }
            }
            updateSession.close();
        } finally {
            if (session != null && session.isOpen()) {
                transaction.commit();
                session.close();
            }
        }

With the   java frameworks like spring and others this code may be even more smaller , like one not needing to  take care of session closing etc . Above code is written in plain java using hibernate. 

Please  try with large data and comment us the results , Also if you have some other better approach to do this please comment . 

Thank You for reading the article

Want to be a better programmer – Read , Read , Read – But How?

If one wants to become a better programmer one thing is for sure that one needs to understand others code.

For beginners when we start writing the code, when using libraries very basic like List , Set , we get stuck in basic things like:

  • What this class do?
  • How to use this class?

Now this comes because we are missing one important trick:

  • First associate a purpose for the class be reading its definition and by name.
  • Now if you have the purpose you will be able to automatically make out what the functions in this class should be
  • Same goes for the functions first associate a purpose.

Now if you start reading the code by understanding the purpose of the class, you will be able to understand and use that class.

Now the next thing you should do is when you understood the purpose is :

  • look at their internal implementations
  • understand the data structure or variables they have declared
  • try to reason about the purpose why they have declared like that
  • Any pors / cons – alternative implementation you can think of.

I bet if one starts doing this in initial part of their programming career, this would help them:

  • to reason lots about other libraries
  • understand new libraries faster
  • also while writing new libraries one will be able to choose right data structures.

Lets see a demo of the above theory:

We all know there is Collections in java which looks somethings like this:

Java Collection Structure

Now lets start first by assigning purpose and see how things follow automatically :

  • Collection Interface – it says one can have some objects inside me.
    • so the functions should be in this class
      • one function to add a object
      • one function to remove a object
  • List Interface – it says its a collection but has a ordering for objects means you get in the order you put
    • so the extra functions in this class should be
      • get on a particular position get(int i)
      • add on a particular position
  • Set Interface – it also says it is a collection but it contains only unique objects and does not care about ordering
    • so the functions in this class
      • one is contains to check whether this already exist as it will not add a object twice

You see by just understanding the class purpose we could easily make out the functions , their purpose.

I am leaving the next part to you guys on looking at the internal implementation. Will make a next blog for this.

Naming is the most important thing

There are only two hard things in Computer Science: cache invalidation and naming things.

This quote is one of my favorite programming quotes and we will discuss the reason why here

We will discuss the naming things part here.

In your entire career in coding what you will do most is reading others code or reading others libraries or new technologies and understanding them, solving bugs in the existing code. If you agree lets move forward.

Now a small story, there was a bug in our existing system and one of my fellow programmer was working on that. The bug was in the core libraries written years back and to add to his woes there was no documentation neither external and nor on code.

He was working on it for days with no result and we were discussing the problem and he said i was unable to understand the code, now i start debugging with him and the bug was solved in minutes (not to brag about my skills), and he said how you are able to go to the problem class and function so easily. I said its just intuition from the naming of the classes and function.

Here i understood its still not natural or important for people the naming convention.

Story part over.

Lets discuss that – what is this intuition thing all about . What i learnt in my coding career that before naming a class first associate a purpose with that class and name the class such that one can understand most part of the purpose with that name and then start adding function in that class and any function if not in alignment to that class purpose then that function should not be there and also through the function name one should be able to make out its implementation without looking at the code.

Now people who read lots of code will realize that’s how most of the good code are written and most of the libraries you will come around will follow this.

Now with the intuition you got from the name without reading the code one can make out the functions and so on. This thing happen in real life as well, if i say something a screen you know mostly what it does (if you think in right context).

Similarly if we start naming our classes or functions correctly this not just help us but help other programmers who will be working on it later.

So be very careful about when you name anything (class , function ) it should convey the purpose.

Also as a developer you should and must build the intution from naming and read code.

Start naming things correctly and read code with the right intuition – dont just jump into the code.

The Solution — Too Much Fine Logging Causing Heap Dump

Many times, we face issues of heap dump in our Java server. Heap increases because your system is generating many objects, and if their lifecycle is long and the system keeps on generating these objects, they will consume all the RAM and heap dump may occur. This may occur due to memory leaks, bad coding, etc.

But these are not the only reasons for heap dump. Once, I faced an issue with our customer where heap dump was occurring regularly, and after debugging, I found that the root cause of the issue was fine logging. You may be surprised how logging can cause heap dump. 

Too much logging can cause heap dump even if they are fine level logs, not info logs. A surprising fact is that fine logging was not enabled.

Why it happened is as follows:

Generally, we log fine logging like in this example:

logger.fine("fine logs ") .

Now, what happens behind the scenes (in the JVM) is that even if fine logs are disabled except for the string “fine logs,” objects will be created as it is passed as an argument if the check for fine logging is inside the fine method.

If there are too many fine logs in the system and there are many threads available doing these operations with lots of fine logs, many char[] objects are created and fill the heap/RAM.

The Solution

There are two ways we should do fine logging.

1. Check the “if” condtion before every fine log, as follows:

if (logger.isFineLoggable()) {
logger.fine("your log ");
}

But, this does not look cool (lots of “if” in the code…)

2. With Java 8, Java gives a new API for logger, which does not take string as an argument; it takes function, which returns string as an argument and evaluates it when this function is called inside the fine method. This way, an object is not created until the function is called.

Here is an example:

logger.fine(() -> "fine logs"); .

With functional programming, the code looks nice and the problem was resolved.

Java – hello world

Here we will write Hello World in java and try to understand basics of how it works.

public class HelloWorld{
    public static void main(String[] args){
         System.out.println("Hello World");
    }

}
  • How to run?
    • Save the above code in a file with name – HelloWorld.java
    • if you are using an IDE like Eclipse/Intellij , right click on file and click run
    • or from command line
      • first run javac HelloWorld.java
      • then run java HelloWorld
  • Now Let’s understand
    • First of all we have described a class here with Name HelloWorld public class HelloWorld
    • In this we described a function with name main as public static void main
    • Main function with this signature is special to java as it calls this main function when your program runs.
    • Now in this main function we have written System.out.println("Hello World");
    • With this line we are telling the System to print the text passed(which is “Hello World” in this case) and system prints this on the command line.