"index files" operation takes too long time.

Julien HENRY

2014-07-25 10:12:29 UTC

Hi Kim,

How many total files do you have in your project?

find /data001/jenkins/sonarqube/local_repository/ma
ke_sonar_project_old/LGEIME4_0 -type f -print | wc -l

The algorithm is to iterate over each file recursively starting from source
dir then decide if file should be indexed:
- first inclusion/exclusion patterns are checked
- language is detected based on file extension -> no language means file
is not indexed

Applying exclusion patterns like **/gen/** and ***/aidl/*** on each file
can be very time consuming. I already think about some potential
optimization, but AFAIK nothing can be done if your pattern starts by a
wildcard.

Another solution in your case it to limit number of files that are
considered.

For example if your project is like:

src1/
src2/
anotherDirWithTonsOfNonJavaFilesThatWillBeIgnoredAnyway/

instead of doing analysis with -Dsonar.sources=. it will be far more
efficient to do -Dsonar.sources=src1,src2

++

Julien

Post by jiyoun.kim
Hi.. I am jiyoun.
I use sonar 4.2.
When I analyze my app(about 150000 lines) for incremental mode,
it takes too long time*(5 minutes)* for âindex filesâ action. So I canât
get any fast feedback L
Bottom is captured log..
SonarQube Runner 2.3
Java 1.6.0_45 Sun Microsystems Inc. (64-bit)
Linux 2.6.35-25-generic amd64
/data001/jenkins/sonarqube/sonar-runner-sonar4.
2/conf/sonar-runner.properties
INFO: Project configuration file: NONE
INFO: Default locale: "en_US", source code encoding: "UTF-8"
/data001/jenkins/sonarqube/local_repository/make_sonar_pro
ject_old/LGEIME4_0/.sonar
INFO: SonarQube Server 4.2
17:55:08.530 INFO - Incremental mode
17:55:08.533 INFO - Load batch settings
17:55:08.615 INFO - User cache: /data001/jenkins/.sonar/cache
17:55:08.623 INFO - Install plugins
17:55:08.634 INFO - Exclude plugins: devcockpit, jira, pdfreport, views,
report , buildstability, scmactivity, buildbreaker
17:55:09.667 INFO - Create JDBC datasource for
jdbc:h2:/data001/jenkins/sonarqu
be/local_repository/make_sonar_project_old/LGEIME4_0/.sonar/.sonartmp/preview140
6278508853-0
17:55:11.493 INFO - Initializing Hibernate
17:55:13.055 INFO - Load project settings
17:55:13.084 INFO - Apply project exclusions
17:55:13.334 INFO - ------------- Scan LGEIME4_0-xt9
17:55:13.338 INFO - Load module settings
17:55:13.585 INFO - Language is forced to java
17:55:13.597 INFO - Loading technical debt model...
17:55:14.021 INFO - Loading technical debt model done: 424 ms
17:55:14.031 INFO - Configure Maven plugins
17:55:14.274 INFO - Compare to previous analysis (2014-07-22)
17:55:14.295 INFO - Compare over 30 days (2014-06-25, analysis of
2014-06-25 17 :43:20.0)
17:55:14.317 INFO - Compare to previous version (2014-07-22)
/data001/jenkins/sonarqube/local_repository/make_
sonar_project_old/LGEIME4_0
/data001/jenkins/sonarqube/local_repository/ma
ke_sonar_project_old/LGEIME4_0/.sonar
/data001/jenkins/sonarqube/local_repository/ma
ke_sonar_project_old/LGEIME4_0
17:55:14.424 INFO - Source encoding: UTF-8, default locale: en_US
17:55:14.424 INFO - Index files
17:55:14.435 INFO - **/gen/**
*17:55:14.435 INFO - **/aidl/***
*18:00:20.534 INFO - 243 files indexed*
18:00:21.023 INFO - Quality profile for java: LGE Java Code Quality
Except Findbugs
18:00:21.238 INFO - Sensor JavaSquidSensor...
18:00:21.333 INFO - Java Main Files AST scan...
18:00:21.336 INFO - 243 source files to be analyzed
==============================================================
JiYoun Kim
Software Infra Team,
Software Capability development Center, CTO, LG Electronics Inc.
Mobile : +82-10-3996-1962
==============================================================