Friday, 28 July 2017

Java 8: Use Lambda to create a Pivot-like result

Lets assume we have a java POJO Person with some fields like year, teamId and sum and we want to create a pivot which shows for every year and teamId the total sum like:
-----------------------------------------------
             |     Test|  Develop|  Deploy|
-----------------------------------------------
     2015|    1450|       5800|       870|
-----------------------------------------------
     2016|    2500|     10000|           0|
-----------------------------------------------
     2017|          0|       3500|       250|
-----------------------------------------------

Define a simple CSV file which keeps the data:
year,teamId,sum
2015,Test,480
2015,Develop,800
2015,Deploy,300
.....
To create our pivot, first we need to read the data in an appropriate structure Map<YearTeam, List<Person>> where YearTeam is also a simple POJO containg year and teamId. For this, Collectors.groupingBy method will really help.
Pattern pattern = Pattern.compile(",");
Map<YearTeam, List<Person>> grouped = new HashMap<>();
try (BufferedReader in = new BufferedReader(new FileReader(fileName));) {
   grouped = in
    .lines()
    .skip(1)
    .map(line -> {
       String[] arr = pattern.split(line);
       return new Person(
                  Integer.parseInt(arr[0]), 
                  arr[1],
                  Integer.parseInt(arr[2]));
       })
    .collect(Collectors.groupingBy(x -> new YearTeam(x.getYear(), x.getTeamId())));
}
Having this map, we want to print first the teams header and then every year with total sum. First step is to obtain the set of teams:
Set<String> teams = map
        .keySet()
        .stream()
        .map(x -> x.getTeamId())
        .collect(Collectors.toCollection(TreeSet::new));

We can define some useful parameters for pretty printing:
public static int COLUMN_WIDTH = 9;
public static String COLUMN_DELIMITER = "|";
public static String LINE_DELIMITER = "-";
And some print utilities methods:
private static void printLineDelimiter(int columns) {
    System.out.println();
    System.out.println(String.join("", Collections.nCopies(
               columns*(COLUMN_WIDTH +1), LINE_DELIMITER)));
}

private static void printTeamsHeader(Set<String> teams) {
    System.out.printf("%" + (COLUMN_WIDTH + 1) + "s", COLUMN_DELIMITER);
    teams.stream().forEach(t -> System.out.printf("%" + (COLUMN_WIDTH + 1) + "s", 
                                t + COLUMN_DELIMITER));
}

private static void printYear(int year) {
    System.out.printf("%" + (COLUMN_WIDTH +1) + "s" , year + COLUMN_DELIMITER);
}

private static void printTotal(long total) {
    System.out.printf("%" + COLUMN_WIDTH + "s", total);
}
With these we can start to print teams header:
int columns = teams.size()+1;

printLineDelimiter(columns);
printTeamsHeader(teams);
printLineDelimiter(columns);
Next we need to print actual data. To compute the total sum we will use Collectors.summingLong :
Set<Integer> years = map
        .keySet()
        .stream()
        .map(x -> x.getYear())
        .collect(Collectors.toSet());

years
    .stream()
    .forEach(y -> {
        printYear(y);
        teams.stream().forEach(t -> {
            YearTeam yt = new YearTeam(y, t);
            List<Person> persons = map.get(yt);
            long total = persons == null ? 0 :
                    persons.stream()
                           .collect(Collectors.summingLong(Person::getSum));
            printTotal(total);
            System.out.print(COLUMN_DELIMITER);
        });
        printLineDelimiter(columns);
    });

Friday, 27 May 2016

Maven Settings Everywhere

Maven can have its settings.xml defined in more ways:

1. Global settings: inside %MAVEN_INSTALL_FOLDER%/conf/
   This file is always used! And you should be aware about this in cases you will also take settings.xml in other way.

2. User settings:  inside %USER_HOME%/.m2/
   This file is optional. Any settings defined in the user settings take precedence over the corresponding global settings.

3. Command line: through --settings %SETTINGS_FILE_PATH%
  
If there is a global settings file, but in some cases you must use another settings file from command line, you should also override the global settings  from command line with:

  --global-settings %SETTINGS_FILE_PATH%

One problem I encountered if you do not do it so is related to mirrors. For example you can have in global settings file:

<mirror>           
    <id>myproject</id>           
    <mirrorOf>*,!myproject.releases</mirrorOf>           
    <url>http://myrepo/</url>       
</mirror>

and inside settings.xml used through command line other mirror like:

<mirror>           
    <id>myproject2</id>           
    <mirrorOf>*,!mydependency</mirrorOf>           
    <url>http://myrepo2/</url>       
</mirror>

and a repository with id = mydependency  which must be resolved from a specific url repo (other than http://myrepo2/).

Because of the mirror in global settings mydependency wont't be resolved by its specific repo, but from the url http://myrepo/ (because of mirror rules) and it will not be found!

Tuesday, 12 April 2016

NextReports: Till Next Time!

        After working for so many years on NextReports I wondered what was it's actual birthday?

        We started on early 2006 to work on it (perhaps first classes where written even at the end of 2005), but the first installer for users was created at the end of the year. Keeping all old releases even far before the first one which hit the shelves as a downloadable on NextReports site (version 2.0 on 6 of March 2009), I found it easily: it was a 0.3.21-beta created on 18 of December 2006. So I guess this is the real birthday date.


        9 years and 4 months have passed from then and so many things happened meanwhile. Year after year a new release was born with new features, new hopes and new wishes. Year after year people
encouraged us to continue and to not give up. And we did. For ten years.

        I look at the first article published on NextReports blog "NextReports : A Reports Tale" and even now I can sense the joy, the unrest, the vision and most of all the simplicity of NextReports.

        Now, the time has passed and the steps will guide us to other horizons. The work is there , our part of life is there and I just hope it will continue to exist long time from now.



NextReports Log :  April 2016

Tuesday, 22 December 2015

Do you reuse JDBC connections? Then be aware.

Sometimes we want to reuse a JDBC connection because the time taken to create a new one can be too big. This is the case we use inside NextReports Designer . Someone connects to a data source and then he can:
  • see tables, views, procedures
  • see columns information 
  • make some validations
  • create and run queries, reports and charts
Because the same connection is reused time after time, it is not closed unless we disconnect from the data source.

If we make many operations on that data source, after some time we can see an error raised like "too many open cursors" . Any database has a property OPEN_CURSORS which specifies the maximum number of open cursors. If this is reached, the error is raised.

In NextReports Designer, this was seen when a chart per record was used inside a report in detail band and after some hundreds of records the charts where not generated anymore because the exception was raised.

When a connection is closed, all the cursors are also closed. But in our case we reuse the connection and no close is done. So how to close the open cursors in JDBC?

The answer resides in calling commit() on our connection. We must take care if autocommit is true. If it is true, calling commit() will raise an exception  so this commit() must be called only in cases when autocommit is false.

Thursday, 26 November 2015

Windows Outlook knocks out Eclipse

I was working on some project inside Eclipse. Suddenly I saw an error on Eclipse project and no clean or rebuild worked anymore. Inspecting the error I saw that my output build folder cannot be deleted because is used by other application.

I closed Eclipse and I tried to delete the folder manually, but Windows could not delete it either. I opened in windows resmon.exe and inside CPU tab / Associated Handles I searched for folder path.
I found an entry with an OUTLOOK handle to that path. Why? Just because I sent a mail with a file from that path.

Evil OUTLOOK is so buggy!

Friday, 2 October 2015

An awkward case of browser scroll

Context 

NextReports Server contains a dashboard were a set of widgets like tables, charts and  many others can be shown. Tables can contain sometimes a lot of data and a pagination & a scroll bar is used in such cases. If many widgets are inside a dashboard, then a browser scroll will be visible.


Problem

A table with pagination & scroll bar will make the entire browser to have a scroll even if there is no need of it. This scroll bar height is as if the table widget is fully rendered with no scroll at all.


 The dashboard should look instead as the following:



Investigation

Looking with firebug inspection to html & css a div element is seen: 
<div style="position: absolute; left: -9999px; width: 1px; height: 1px;">
  <input id="ide19focus" type="hidden" value="" name="ide19focus">
  <input type="submit">
</div>
This is not created by developers. Because Wicket framework is used, this html code is added by Pagination Navigator and it represents a non visible submit button used by navigation links. By commenting css attributes of this div style it can be seen the normal functionality.


Solution

The entire widget is contained inside a div with a class like: 
<div class="tableWidgetView">
 To make our entire dashboard look ok, tableWidgetView class must contain in addition a single css attribute:  

          position: relative;  

That's why because the problematic div (found inside tableWidgetView div) has an absolute position which is referring its parent. Simple solution, but hard to find.

Tuesday, 11 August 2015

Competition: Creating an application just for fun

When you're a developer and you like what you are doing, at some moment in time you will create something just for fun.

In 2012 I found myself in such a situation. I was playing with my friends to guess scores from Premier League. Till that time we kept an excel file with all the users, their scores and we had a special column to compute a total for all games in a stage (1 point for guessed prognostic and 3 points for exactly guessed score). That file was kept by one of us, sent by mail to all to complete scores, then sent back, and the admin user was updating the main file and so on. This process became soon a pain.

I started to create a web application to make it easier for all of us. Having an online web application with user security where users log and complete scores, then every computation is automatically done, was the right thing to do.

I did not want just a simple competition, like our Premier League. So I thought at it in a general manner. Administrator should be able to create a competition with a number of teams, stages and games, then a number of users will be registered to it. When a stage is approaching, users that did not complete it will be notified by mail to do so. In the day of the stage, application will inquire a site for live scores and, when finish, the scores will be automatically saved and a process of points computation will be run.

I chose as user scoring a simple algorithm to give 1 point for guessed prognostic, 3 points for guessed score when were at most 3 goals, n points for guessed score were n > 3 was the number of goals. This decision brings benefit to those that will guess a 2-2, 3-2 , or any score with more than 3 goals. To make it even more interesting, I created a section of questions for a competition. If user guesses a response he will be rewarded with 3 points. In time, another feature was added: a playoff. A competition can have a playoff phase. Starting from a defined stage, users will compete one against other and the user with a bigger scoring in that stage will advance to next round. The winner of the playoff will get a bonus of 10 points.

Using Java, Wicket and Spring I started to make this application happen. Spring Security helped to achieve users login/logout feature. When users are not logged , they can only see the real scores, but cannot see any data related to a user. When users are logged, they can see their scores, and after the stage started, they can see others scores.

Main screen is just a list of competitions (with Top3 aside) which by default shows only 4 of them. To look for other competitions, a search feature is used.


For small window sizes (like on mobile phones) the main screen is adjusted accordingly:



Admin user will be the only one that will see a toolbar inside main screen. From here, he can create users, competitions, teams, stages, games, questions. Also, admin can manually run the computation process , can reset top trends, can import a competition from a text file and can see the logins history.

By clicking a competition, a new screen will show all data for that competition:


At top, an auto-scroll banner will show the scores for last stage. User can see any stage from that competition and can enter scores for any stage that did not start yet. When the scores are automatically entered and after the computation process, user will see the number of points gained for every game and a total for that stage. User can also see how all other users performed. A Top section will show the total points for that competition with trend arrows (who climbed, who went down). There are a lot of other sections like the top of the real teams from that competition, the playoff, a rss feed to find news about that competition, statistics, graphics, questions.

Inside Statistics section, a general report for entire competition can be generated for every user. User can also see who was his lucky team because it brought him the most points. Also average points per stage top and the best stage scoring can be seen here:


Graphics can show user points evolution and some "awards" like who is the luckiest, who is the best at home games, who is the best at away games and who is the best at draw games.


All reports and charts were created and run using NextReports . I also made the code available on github.

There are more than 3 years from it's inception and many editions of Premier League, UEFA Champions League, World Cup, European Championship were finished. Competition fulfilled our needs: very easy to manage, to enter scores, notifications, auto computation processes. At the end all we need to do is to buy a cup for the winner and to celebrate him. Hurray!