Minutes from the workshop 

excellencemapping.net

• The quality of data needs to be improved: more emphasis on data cleaning and disambiguation
• Different publication types–proceedings, journals, books–need to be uniformly covered in bibliometric data to have a better basis for comparison of different fields and measurement of interdisciplinary impact
• More attention towards highligting the differences of citations meaning in different fields needed
• Citations are not the only measure of quality, which is a general practice when ranking universities
• A counter-example of Netherlands, where evaluation of research projects incorporates social impact of research
• Performance of research groups and individuals could incorporate resources available, i.e. measures of productivity and not total output.

• Larger coverage of scientific literature can possibly solve the issues coming from differences in citation cultures
  • Coverage in databases is particularly difficult for social sciences, due to difference in forms of scientific communication
  • Geographic differences in citation cultures
  • Different publication languages not covered homogeneously
• Effects of publication format/practical restriction of citations by journal format
  • Values of citations differ based on number of references in the citing paper

• Institutions may actually not be interested in the attention that someone gets, but in the output that is produced given the funding provided
• The mean should not be used with bibliometric data, as is not an appropriate measure of "average"–and often is not defined–for broad distributions, which are commonly met
• "Altmetric" is open to provide data
• In many contexts related to measuring scientific performance we need correct for attention, and not take it as a proxy of performance.

• che-ranking.de
• u-multirank.eu

• Awareness about the risks of reputation surveys: visibility bias, geographic scope; However, high response rate (professors surveyed)
  • Collect all professors at universities

• topuniversities.com/university-rankings

• Academic reputation surveys (QS academic reputation data set) provided
• In order to interpret results, institutions need to get insights into the methodology
• Some ranking providers are more open than others
• Reinforcing feedback of rankings to reputation surveys is probably pronounced
• Variability in data has to be taken into account
  • Australia case study: ranking not possible, because the variability of opinions within the institutions is higher than across them

• vosviewer.com
• leidenranking.com

• Full vs. fractional counting: full counting is biased towards fields with lots of collaborations
• Known: collaborative publications tend to be cited more often than non-collaborative publications
• Use of appropriate bibliometric methodology is important
  • Percentile-based indicators
  • No blind use of databases, removing non-core journals
• The weights for fractional counting determined based on the number of different institutions involved
• Fractional counting at the individual level not recommended
  • At the aggregate level some effects cancel out, that may be a problem at the individual level
• Extensive manual cleaning of affiliation data performed
• Thomson Reuters, the owner of Web of Science, works with universities to introduce canonical affiliation names
  • So far for 4800 university names disambiguated

• Misbehaviour of reviewers can take very creative forms: not only nonobjective rejections, but prolongation of the review process and other ways to keep control over the paper
• While metrics become more popular, they influence the dynamics of science, but this influence can be beneficial in certain cases
  • In Italy, associations played a major role previously
• The system of science needs to be steered towards beneficial external motivations for scientists
• Mechanisms reinforcing the social norms that restrict the "competitive spirits" needed

• Suggestion made to Altmetric to not report an aggregate score
• The real implication and usage of altmetics for measurements in science not clear yet
• Measures incorporating altmetrics have to very robust, as these are prone to manipulations, e.g. by means of link farms
• Altmetrics don't measure social impact, but more the social media attention, which cannot be used as scientific advancement measure

• physauthorsrank.org

• Interesting to retrospectively compare predictions of SARA and the prizes won later by the top in the ranking
• The approach of using historical data for predictions assumes that the dynamics don't change which is not necessarily the case
• Yearly contest idea: Predict scientific prize winners based on scholarly citation data

• mapequation.org

• Zero-, first-, second-order Markov models can complement each other, thus can have higher predictive power when used together

• Productive author probably has more collaborations (thus, more chances to be more central) and higher chance to get a paper to be in top 10% by random
• Self-citations are considered in the study, excluding them can be considered
• The citation success is predicted for a paper, and the more central author contributes to the prediction
  • Corollary is that a young researchers citation success is dependent on the PI's centrality, if the latter is in the authors list
• The model can be of interesting for journal editors, for the data providers to check this social bias based on their data.
• The levels of social bias among different fields can be compared
  • For computer science (used in the study) it is expected to be higher social bias is higher, than in e.g. physics due to the conferences (personal meetings) and conference proceedings being the main types of communication.

• webofknowledge.com

• Request for a standard test data set with disambiguated and normalized institution names for research purposes
• From a data scientist's perspective not easy to decouple the effects from sudden addition of large numbers of journals from the changes in the systems dynamics, thus providing data on the journal inclusion time can be helpful
• About 3,000 OA journals indexed
• A work in progress showing the citation differences between open access and subscription-based journals
• PLoS One: 83000 articles between 2004-2013
• Normalization on journal level subject categories has the disadvantage when considering journals with broad topic coverage
• The historical data for newly included journals also indexed

• springer.com/citations

• Estimation that 80% will be open access
• Possible emergence of two main journal categories: big container journals versus flagship journals
• Although the impact factor for container journals is not meaningful, it has an influence on behaviour; People from low average IF fields jump onto wagon of PLoS to benefit from the IF coming from life sciences
• Although download data possibly has potential for quantifying impact, information on total downloads not trivial to obtain, as usually the publications are hosted on multiple services with different data sharing policies.
• Social media such as Twitter can be considered as a new fast communication channel for science, but at the same time it can and is used for "advertising" purposes

• Leader detection method can be applied to detect authors cultivating interdisciplinarity, facilitating knowledge flow between disciplines
• For a better evidence for the fitness model, attempts to consider the fitness parameter as external and estimate it outside of the model's scope

• Deeper investigation of relation between downloads and citations